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i  GAO  MILITARY  READINESS 


DOD  Needs  to  Strengthen  Management  and  Oversight 
of  the  Defense  Readiness  Reporting  System 

Highlights  of  GAO-09-518,  a  report  to  the 
Subcommittee  on  Readiness  and 
Management  Support,  Committee  on 
Armed  Services,  U.S.  Senate 


Accountability  Integrity  Reliability 

Highlights 


Why  GAO  Did  This  Study 

The  Department  of  Defense  (DOD) 
reports  data  about  the  operational 
readiness  of  its  forces.  In  1999, 
Congress  directed  DOD  to  create  a 
comprehensive  readiness  system 
with  timely,  objective,  and  accurate 
data.  In  response,  DOD  started  to 
develop  the  Defense  Readiness 
Reporting  System  (DRRS).  After  7 
years,  DOD  has  incrementally 
fielded  some  capabilities,  and, 
through  fiscal  year  2008,  reported 
obligating  about  $96.5  million. 

GAO  was  asked  to  review  the 
program  including  the  extent  that 
DOD  has  (1)  effectively  managed 
and  overseen  DRRS  acquisition  and 
deployment  and  (2)  implemented 
features  of  DRRS  consistent  with 
legislative  requirements  and  DOD 
guidance.  GAO  compared  DRRS 
acquisition  disciplines,  such  as 
requirements  development,  test 
management,  and  DRRS  oversight 
activities,  to  DOD  and  related 
guidance,  and  reviewed  the 
system’s  current  and  intended 
capabilities  relative  to  legislative 
requirements  and  DOD  guidance. 
We  did  not  evaluate  DOD’s  overall 
ability  to  assess  force  readiness  or 
the  extent  that  readiness  data 
reflects  capabilities,  vulnerabilities, 
or  performance  issues. 


What  GAO  Recommends 


GAO  is  making  recommendations 
to  address  the  risks  facing  DOD  in 
acquiring  and  developing  DRRS 
and  increase  the  chance  of  success. 
DOD  agreed  or  partially  agreed 
with  three  of  our  eight 
recommendations,  and  disagreed 
with  the  remaining  five  because  it 
stated  that  it  was  already  taking 
actions  in  these  areas. 

View  GAO-09-518  or  key  components. 

For  more  information,  contact  Sharon  Pickup 
at  (202)  512-9619  or  pickups@gao.gov,  or 
Randolph  C.  Hite  at  (202)  512-3439  or 
hiter@gao.gov . 


What  GAO  Found 

DOD  has  not  effectively  managed  and  overseen  the  DRRS  acquisition  and 
deployment,  in  large  part  because  of  the  absence  of  rigorous  and  disciplined 
acquisition  management  controls  and  an  effective  governance  and 
accountability  structure  for  the  program.  In  particular,  system  requirements 
have  not  been  effectively  developed  and  managed.  For  example,  user 
participation  and  input  in  the  requirements  development  process  was,  until 
recently,  limited,  and  requirements  have  been  experiencing  considerable 
change,  are  not  yet  stable,  and  have  not  been  effectively  controlled.  In 
addition,  system  testing  has  not  been  adequately  performed  and  managed.  For 
example,  test  events  for  already  acquired  system  increments,  as  well  as 
currently  deployed  and  operating  increments,  were  not  based  on  well-defined 
plans  or  structures,  and  test  events  have  not  been  executed  in  accordance 
with  plans  or  in  a  verifiable  manner.  Moreover,  DRRS  has  not  been  guided  by 
a  reliable  schedule  of  work  to  be  performed  and  key  activities  to  occur.  These 
program  management  weaknesses  can,  in  part,  be  attributed  to  long-standing 
limitations  in  program  office  staffing  and  program  oversight  and 
accountability.  Despite  being  a  DOD-wide  program,  until  April,  2009  DRRS 
was  not  accountable  to  a  DOD-wide  oversight  body,  and  it  was  not  subject  to 
DOD’s  established  mechanisms  and  processes  for  overseeing  business 
systems.  Collectively,  these  acquisition  management  weaknesses  have 
contributed  to  a  program  that  has  fallen  well  short  of  expectations,  and  is 
unlikely  to  meet  future  expectations. 

DOD  has  implemented  DRRS  features  that  allow  users  to  report  certain 
mission  capabilities  that  were  not  reported  under  the  legacy  system,  but  these 
features  are  not  fully  consistent  with  legislative  requirements  and  DOD 
guidance;  and  DOD  has  not  yet  implemented  other  features.  The  geographic 
combatant  commands  are  currently  reporting  their  capabilities  to  execute 
most  of  their  operations  and  major  war  plans  in  DRRS,  and  DOD  is  reporting 
this  additional  information  to  Congress.  However,  because  DRRS  does  not  yet 
fully  interface  with  legacy  systems  to  allow  single  reporting  of  readiness  data, 
the  military  services  have  not  consistently  used  DRRS’s  enhanced  capability 
reporting  features.  For  example,  as  of  May  2009,  the  Army  and  Navy  had 
developed  interfaces  for  reporting  in  DRRS,  while  the  Marine  Corps  required 
units  to  only  report  in  their  legacy  system.  Recently,  the  Marine  Corps  also 
began  developing  an  interface  and  has  done  limited  reporting  in  DRRS.  In 
addition,  DRRS  has  not  fully  addressed  the  challenges  with  metrics  that  led 
Congress  to  require  a  new  readiness  reporting  system.  DRRS  metrics  are  less 
objective  and  precise,  and  no  more  timely  than  the  legacy  system  metrics. 
Users  have  also  noted  that  DRRS  lacks  some  of  the  current  and  historical  data 
and  connectivity  with  DOD’s  planning  systems  necessary  to  manage  and 
deploy  forces.  Until  these  limitations  are  fully  addressed,  DRRS  will  not  have 
the  full  complement  of  features  necessary  to  meet  legislative  and  DOD 
requirements,  and  users  will  need  to  rely  on  legacy  reporting  systems  to 
support  mission-critical  decisions. 
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Abbreviations 


CJCSI 

DIO 

DOD 

DRRS 

ESORTS 

GSORTS 

JITC 

MAIS 

OSD 

SIPRNet 

TEMP 

USD  (P&R) 


Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 

DRRS  Implementation  Office 

Department  of  Defense 

Defense  Readiness  Reporting  System 

Enhanced  Status  of  Resources  and  Training  System 

Global  Status  of  Resources  and  Training  System 

Joint  Interoperability  Test  Command 

Major  Automated  Information  System 

Office  of  the  Secretary  of  Defense 

Secure  Internet  Protocol  Router  Network 

Test  and  Evaluation  Master  Plan 

Under  Secretary  of  Defense  for  Personnel  and 

Readiness 


This  is  a  work  of  the  U  .S .  government  and  is  not  subject  to  copyright  protection  in  the 
United  States.  The  published  product  may  be  reproduced  and  distributed  in  its  entirety 
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copyrighted  images  or  other  material,  permission  from  the  copyright  holder  may  be 
necessary  if  you  wish  to  reproduce  this  material  separately. 
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GAO 

^^Accountability  *  Integrity  *  Reliability 

United  States  Government  Accountability  Office 
Washington,  DC  20548 


September  25,  2009 

The  Honorable  Evan  Bayh 
Chairman 

The  Honorable  Richard  Burr 
Ranking  Member 

Subcommittee  on  Readiness  and  Management  Support 
Committee  on  Armed  Services 
United  States  Senate 

To  assess  the  ability  of  U.S.  forces  to  execute  the  wartime  missions  for 
which  they  were  designed,  as  well  as  other  assigned  missions,  and  to  make 
decisions  about  deploying  forces,  the  Department  of  Defense  (DOD)  relies 
heavily  on  readiness  information  derived  from  multiple  information 
systems.  Over  the  years,  we  and  others  have  identified  shortcomings  in 
DOD’s  readiness  assessment  and  reporting,  such  as  limitations  in  the 
completeness  and  precision  of  readiness  data  and  a  tendency  to  focus  on 
examining  the  status  of  personnel,  equipment,  and  other  resources  rather 
than  broader  capabilities.  Congress  addressed  DOD’s  readiness  reporting 
in  the  Strom  Thurmond  National  Defense  Authorization  Act  for  Fiscal  Year 
19991  by  adding  section  117  to  Title  10  of  the  U.S.  Code,  directing  the 
Secretary  of  Defense  to  establish  a  comprehensive  readiness  reporting 
system  to  measure,  in  an  “objective,  accurate,  and  timely  manner,”  the 
capability  of  the  armed  forces  to  carry  out  the  National  Security  Strategy 
prescribed  by  the  President,  the  defense  planning  guidance  provided  by 
the  Secretary  of  Defense,  and  the  National  Military  Strategy  prescribed  by 
the  Chairman  of  the  Joint  Chiefs  of  Staff. 

In  June  2002,  the  Deputy  Secretary  of  Defense  directed2  the  Under 
Secretary  of  Defense  for  Personnel  and  Readiness  (USD  P&R)  to  develop, 
field,  maintain,  and  fund  the  Enhanced  Status  of  Resources  and  Training 
System  (ESORTS),  which  is  the  automated  readiness  reporting  system 


’Pub.  L.  No.  105-261,  §373  (1998).  Codified  at  10  U.S.C.  §117. 

department  of  Defense  Directive  7730.65,  Department  of  Defense  Readiness  Reporting 
System  (DRRS)  (June  3,  2002). 
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within  the  Defense  Readiness  Reporting  System  (DRRS).3  He  also  directed 
that  DRRS  build  upon  existing  processes  and  readiness  assessment  tools 
to  establish  a  capabilities-based,  adaptive,  near-real-time  readiness 
reporting  system.  In  addition,  in  June  2004,  the  Secretary  of  Defense 
directed  USD  (P&R)  to  develop  DRRS  in  a  manner  that  would  support  the 
data  requirements  of  various  users  of  readiness  information,  such  as  the 
Chairman  of  the  Joint  Chiefs  of  Staff,  the  combatant  commanders,  the 
Secretaries  of  the  military  departments,  and  the  Chief  of  the  National 
Guard  Bureau,  including  their  requirements  for  data  on  the  availability, 
readiness,  deployment,  and  redeployment  of  forces.4 

USD  (P&R)  established  a  DRRS  Implementation  Office  (DIO)  to  manage 
the  system’s  acquisition,  including  managing  system  development  and 
engaging  the  user  community.  The  DIO  has  used  support  contractors  to 
develop  the  system  and  reported  obligating  about  $96.5  million  for  DRRS 
from  fiscal  year  2001  through  fiscal  year  2008.  Some  fielded  system 
capabilities  are  currently  being  used  to  varying  degrees  by  the  user 
community.  The  DIO  originally  estimated  that  DRRS  would  achieve  full 
operational  capability  in  fiscal  year  2007,  but  currently  expects  DRRS  to 
reach  full  capability  in  2014.  In  September  2008,  the  DIO  projected  that  it 
would  spend  about  $135  million  through  fiscal  year  2014. 

Recognizing  that  DRRS  was  not  yet  fully  deployed  or  operational  and  in 
light  of  our  prior  work  on  readiness-related  issues,  you  asked  us  to  review 
DOD’s  efforts  to  develop  and  implement  DRRS,  including  the  program’s 
status,  and  the  extent  that  DRRS  addresses  the  challenges  that  led 
Congress  to  require  a  new  system,  such  as  the  availability  of  information 


3ESORTS  is  an  automated  readiness  reporting  tool  designed  to  collect  capability  and 
resource  data,  while  DRRS  is  the  broader  system  that,  according  to  the  2002  DRRS 
Directive,  measures  and  reports  on  the  readiness  of  military  forces  and  the  supporting 
infrastructure  to  meet  missions  and  goals  assigned  by  the  Secretary  of  Defense.  However, 
ESORTS  is  now  viewed  as  a  tool  within  DRRS. 

4Secretary  of  Defense  Memorandum,  Policy  Implementation  to  Establish  Commander, 
USJFCOM  (CDRUSJFCOM),  as  the  Primary  Joint  Force  Provider  (JFP)  (June  25,  2004). 
The  U.S.  military  organizes  its  global  presence  into  a  series  of  geographic  and  functional 
combatant  commands.  The  geographic  combatant  commands — U.S.  Central  Command, 

U.S.  European  Command,  U.S.  Northern  Command,  U.S.  Pacific  Command,  U.S.  Southern 
Command,  and  U.S.  Africa  Command  — have  authority  over  all  U.S.  military  forces 
operating  within  a  specified  area  of  operation  and  are  directly  responsible  for  the 
performance  of  missions  assigned  to  the  command.  The  functional  combatant 
commands — U.S.  Joint  Forces  Command,  U.S.  Special  Operations  Command,  U.S.  Strategic 
Command,  and  U.S.  Transportation  Command — possess  worldwide  functional 
responsibilities,  such  as  joint  training,  force  provision,  and  global  command  and  control. 
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on  capabilities,  and  the  precision,  timeliness,  reliability,  and  objectivity  of 
readiness  metrics.  In  addressing  these  issues,  we  assessed  the  extent  to 
which  (1)  DOD  has  effectively  managed  and  overseen  DRRS  acquisition 
and  deployment,  and  (2)  features  of  DRRS  have  been  implemented  and  are 
consistent  with  legislative  requirements  and  DOD  guidance. 

To  determine  the  extent  that  DOD  has  effectively  managed  and  overseen 
DRRS  acquisition  and  deployment,  we  analyzed  a  range  of  program 
documentation  and  interviewed  cognizant  program  and  contractor 
officials  relative  to  the  following  acquisition  management  disciplines: 
requirements  development  and  management,  test  management,  schedule 
reliability,  and  human  capital.  For  each  discipline,  we  compared  key 
program  documentation,  such  as  a  requirements  management  plan,  test 
and  evaluation  master  plans,  and  test  reports  to  relevant  DOD,  federal,  and 
related  guidance,  and  we  analyzed  a  statistical  sample  of  individual 
requirements  and  test  cases  to  determine  consistency  among  them.  In 
addition,  we  attended  meetings  of  organizations  established  to  monitor  or 
govern  DRRS  development  and  reviewed  information  from  meetings  that 
we  did  not  attend  and  interviewed  officials  associated  with  these 
meetings. 

To  determine  the  extent  to  which  the  features  of  DRRS  have  been 
implemented  and  are  consistent  with  legislative  requirements  and  DOD 
guidance,  we  reviewed  criteria  such  as  the  legislation  that  mandated  a 
comprehensive  DOD  readiness  reporting  system,  the  DOD  directive  that 
established  DRRS,  program  documentation  and  USD  (P&R)  guidance 
memorandums,  DIO  briefings  to  the  readiness  community,  other 
departmental  instructions,  and  directives  and  memorandums  related  to 
DRRS  requirements  and  implementation.  From  these  documents,  we 
identified  desired  features  of  DRRS  and  compared  them  to  documentary 
and  testimonial  evidence  concerning  system  performance  during  meetings 
with  a  full  range  of  officials  responsible  for  developing  and  using  the 
system.  To  obtain  the  developer’s  perspective,  on  numerous  occasions 
throughout  our  review  we  met  with  officials  from  USD  (P&R),  the  DIO, 
and  the  three  current  DRRS  contractors.  To  obtain  user  perspectives,  we 
met  with  and  surveyed  by  questionnaire  officials  from  the  Joint  Staff,  the 
geographic  and  functional  combatant  commands,  and  the  services,  and 
also  met  with  officials  from  USD  (P&R).  We  also  attended  meetings  of 
organizations  established  to  monitor  or  govern  DRRS  development  and 
analyzed  information  from  meetings  that  we  did  not  attend.  We  also 
directly  observed  the  system’s  capabilities  through  our  own  limited  use  of 
the  system  and  by  observing  others  using  the  system. 
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We  did  not  evaluate  the  department’s  overall  ability  to  assess  the  readiness 
of  its  forces  or  the  extent  that  data  contained  in  any  of  its  readiness 
reporting  systems,  including  DRRS  and  GSORTS,  reflect  capabilities, 
deficiencies,  vulnerabilities,  or  performance  issues.  Our  review  focused  on 
acquisition  and  program  management  issues,  such  as  requirements 
management,  schedule  and  human  capital  requirements,  the  current  usage 
of  DRRS,  and  the  extent  to  which  DRRS’  features  address  legislative 
requirements  and  DOD  guidance. 

We  conducted  this  performance  audit  from  April  2008  through  August 
2009  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives.  Additional  details  on  our 
scope  and  methodology  are  in  appendix  I. 


Background 


Historically,  DOD  has  used  its  readiness  assessment  system  to  assess  the 
ability  of  units  and  joint  forces  to  fight  and  meet  the  demands  of  the 
national  security  strategy.  DOD’s  readiness  assessment  and  reporting 
system  is  designed  to  assess  and  report  on  military  readiness  at  three 
levels — (1)  the  unit  level;  (2)  the  joint  force  level;  and  (3)  the  aggregate,  or 
strategic,  level.  Using  information  from  its  readiness  assessment  system, 
DOD  prepares  and  sends  legislatively  mandated  Quarterly  Readiness 
Reports  to  Congress.  DRRS  is  DOD’s  new  readiness  reporting  system  that 
is  intended  to  capture  information  from  the  previous  system,  as  well  as 
information  about  organizational  capabilities  to  perform  a  wider  variety  of 
missions  and  mission  essential  tasks.  DRRS  is  also  intended  to  capture 
readiness  information  from  defense  agencies  and  installations,  which  were 
not  required  to  report  under  the  previous  system.  Some  DRRS  features  are 
currently  fielded  and  being  used  to  varying  degrees  by  the  user 
community. 


DOD  Collects  a  Wide 
Range  of  Readiness 
Information  to  Support 
Decision  Makers  Within 
and  Outside  DOD 


Laws,  directives,  and  guidance,  including  a  DOD  directive,  Chairman  of 
the  Joint  Chiefs  of  Staff  Instruction  (CJCSI),  Secretary  of  Defense  and 
USD  (P&R)  memorandums,  and  service  regulations  and  messages,  show 
that  readiness  information  and  data  are  needed  to  support  a  wide  range  of 
decision  makers.  These  users  of  readiness  data  include  Congress,  the 
Secretary  of  Defense,  the  Chairman  of  the  Joint  Chiefs  of  Staff,  the 
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combatant  commanders,  the  Secretaries  of  the  military  departments,  and 
the  Chief  of  the  National  Guard  Bureau. 

The  directives  and  guidance  also  list  roles  and  responsibilities  for 
collecting  and  reporting  various  types  of  readiness  data.  For  example, 
CJCSI  3401.02A5  assigns  the  service  chiefs  responsibility  for  ensuring 
required  global  status  of  resources  and  training  system  (GSORTS)  reports 
are  submitted.  GSORTS  is  DOD’s  legacy,  resource-based  readiness 
reporting  system  that  provides  a  broad  assessment  of  unit  statuses  based 
on  units’  abilities  to  execute  the  missions  for  which  they  were  organized 
or  designed  as  well  as  the  current  missions  for  which  they  may  be 
employed.  The  information  in  the  required  GSORTS  reports  includes  units’ 
abilities  to  execute  the  missions  for  which  they  were  organized  or 
designed,  as  well  as  the  status  of  their  training,  personnel,  and  equipment.6 
In  addition,  DOD  directive  7730.65,  which  established  DRRS  as  DOD’s  new 
readiness  reporting  system,  assigns  the  Secretaries  of  the  military 
departments  and  the  commanders  of  the  combatant  commands 
responsibilities  for  developing  mission  essential  tasks  for  all  of  their 
assigned  missions.7 


Requirements  and 
Intended  Characteristics  of 
DOD’s  New  Readiness 
Reporting  System 


Prior  to  1999,  we  identified  challenges  with  DOD’s  existing  readiness 
reporting  system,  GSORTS,  and  in  1999,  Congress  directed  the  Secretary 
of  Defense  to  establish  a  comprehensive  readiness  reporting  system.8  The 
legislation  requires  the  system  to  measure  in  an  objective,  accurate,  and 
timely  manner  the  capability  of  the  armed  forces  to  carry  out  (1)  the 
National  Security  Strategy  prescribed  by  the  President,  (2)  the  defense 


5Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3401. 02A,  Global  Status  of  Resources  and 
Training  System,  (GSORTS)  (Feb.  27,  2004). 

6A11  combat,  combat  support,  and  combat  service  support  units  that  have  the  potential  to 
support  an:  operations,  contingency,  or  single  integrated  operational  plan;  or  a  contingency 
operation  are  required  to  report. 

'A  mission  is  a  task  or  a  series  of  tasks  with  a  purpose.  Mission  essential  tasks  are  those 
tasks  that  are  absolutely  necessary,  indispensable,  or  critical  to  mission  success.  DRRS’ 
capability  data  are  based  on  commanders’  assessments  of  their  organizations’  capabilities 
to  carry  out  their  missions  and  mission  essential  tasks. 

sThe  Strom  Thurmond  National  Defense  Authorization  Act  for  Fiscal  Year  1999,  Pub.  L.  No. 

105- 261  §373  (1998),  codified  at  10  U.S.C.  §117.  Section  373  initially  directed  the  Secretary 
to  establish  and  implement  the  readiness  reporting  system  no  later  than  January  15,  2000. 
An  amendment  in  the  National  Defense  Authorization  Act  for  Fiscal  Year  2000,  Pub.  L.  No. 

106- 65,  §361  changed  the  date  from  January  15,  2000,  to  April  1,  2000. 
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planning  guidance  provided  by  the  Secretary  of  Defense,  and  (3)  the 
National  Military  Strategy  prescribed  by  the  Chairman  of  the  Joint  Chiefs 
of  Staff. 

To  address  the  requirements  established  by  Congress,  the  Office  of  the 
Deputy  Under  Secretary  of  Defense  (Readiness)  began  in  2001  to  build 
consensus  among  DOD’s  senior  readiness  leaders  for  an  improved 
readiness  assessment  system.  For  example,  the  Deputy’s  office  distributed 
a  list  of  key  characteristics  of  the  improved  readiness  assessment  system 
to  the  leaders  in  advance  of  scheduled  meetings.  The  system’s  key  desired 
characteristics  included  allowing  near-real-time  access  to  readiness  data 
and  trends,  enabling  rapid,  low-cost  development  using  classified  Internet 
technology,  and  reducing  the  reporting  burdens  on  people.  Since  then 
various  directives  and  memorandums  have  been  issued  regarding  DRRS 
responsibilities,  requirements,  and  related  issues.  For  example: 

•  On  June  3,  2002,  the  Deputy  Secretary  of  Defense  established  DOD’s 
new  readiness  reporting  system,  as  directed  by  Congress,  by  signing 
DOD  Directive  7730.65.  According  to  this  directive,  DRRS  is  intended 
to  build  upon  DOD’s  existing  processes  and  readiness  assessment  tools 
to  establish  a  capabilities-based,  near-real-time  readiness  reporting 
system.  The  DRRS  directive  assigned  USD  (P&R)  responsibilities  for 
developing,  fielding,  maintaining,  and  funding  ESORTS  (the  tool  to 
collect  capability,  resource,  and  training  information)  and  overseeing 
DRRS  to  ensure  accuracy,  completeness,  and  timeliness  of  its 
information  and  data,  its  responsiveness,  and  its  effective  and  efficient 
use  of  modern  practices  and  technologies.  In  addition,  the  USD  P&R  is 
responsible  for  ensuring  that  ESORTS  information,  where  appropriate, 
is  integrated  into  DOD’s  planning  systems  and  processes.  The  directive 
also  states  that  until  ESORTS  becomes  fully  operational,  the  Chairman 
of  the  Joint  Chiefs  of  Staff  shall  maintain  the  GSORTS  database. 

•  On  June  25,  2004,  the  Secretary  of  Defense  issued  a  memorandum, 
which  directed  USD  (P&R)  to  develop  DRRS  to  support  data 
requirements  identified  by  the  Chairman  of  the  Joint  Chiefs  of  Staff,  the 
combatant  commanders,  the  Secretaries  of  the  Military  Departments, 
and  the  Chief,  National  Guard  Bureau  to  include  availability,  readiness, 
deployment,  and  redeployment  data.9 


Secretary  of  Defense  Memorandum,  Policy  Implementation  to  Establish  Commander, 
USJFCOM  ( CDRUSJFCOM ),  as  the  Primary  Joint  Force  Provider  (JFP),  (June  25,  2004). 
The  memorandum’s  primary  purpose  was  to  direct  the  Commander  of  the  U.S.  Joint  Forces 
Command  to  assume  the  duties  of  DOD’s  primary  joint  force  provider. 
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•  On  November  2,  2004,  USD  (P&R)  issued  a  DRRS  interim 
implementation  guidance  memorandum. 10  In  this  memorandum,  the 
undersecretary  noted  that  he  had  established  a  DIO  to  provide 
reporting  assistance  for  units.  The  memorandum  also  stated  that 
combatant  commanders  would  begin  reporting  readiness  by  mission 
essential  tasks  by  November  30,  2004.  The  memorandum  also  directed 
the  services  to  develop  detailed  implementing  guidance  for  reporting 
and  assessing  mission  essential  task  readiness  in  ESORTS  within  their 
respective  services,  and  set  a  goal  for  the  services  to  implement  the 
mission  essential  task  reporting  process  by  September  30,  2005.  To 
meet  these  mission  essential  task  reporting  requirements,  USD  (P&R) 
directed  commanders  to  rate  their  organizational  capabilities  as  (1)  yes 
or  “Y”,  (2)  qualified  yes  or  “Q”,  or  (3)  no  or  “N.”  A  “Y”  indicates  that  an 
organization  can  accomplish  the  rated  tasks  or  missions  to  prescribed 
standards  and  conditions  in  a  specified  environment.  It  should  reflect 
demonstrated  performance  in  training  or  operations.  A  “Q”  indicates 
that  performance  has  not  been  demonstrated,  and,  although  data  may 
not  readily  support  a  “Y,”  the  commander  believes  the  organization  can 
accomplish  the  rated  task  or  mission  to  standard  under  most 
conditions.  An  “N”  indicates  that  an  organization  cannot  accomplish 
the  rated  task  or  mission  to  prescribed  standards  in  the  specified 
environment  at  the  time  of  the  assessment. 

•  The  November  2004  memorandum  also  stated  that  the  expected 
transition  from  GSORTS  to  ESORTS  was  scheduled  to  begin  in  fiscal 
year  2005.  According  to  the  2004  memorandum,  the  ESORTS  module  of 
DRRS  would  provide,  among  other  things,  visibility  of  the  latest 
GSORTS  information  reported  by  units,  and  detailed  resource 
information  from  authoritative  data  sources  with  the  capability  to 
aggregate  or  separate  the  data.  This  memorandum  signaled  a  change  in 
program  direction.  Although  the  2002  DOD  directive  stated  that  DRRS 
is  intended  to  build  upon  DOD’s  existing  processes  and  readiness 
assessment  tools,  the  2004  memorandum  indicated  that  DRRS  was  to 
replace  GSORTS,  as  the  ESORTS  module  of  DRRS  captured  both 
capabilities  and  resource  data. 


10Under  Secretary  of  Defense  for  Personnel  and  Readiness  Memorandum,  Department  of 
Defense  Readiness  Reporting  System  (DRRS)  Interim.  Implementation  Guidance  (Nov.  2, 
2004).  USD  (P&R)  issued  three  subsequent  memorandums  between  August  10,  2005,  and 
August  23,  2006,  which  expanded  and  clarified  reporting  requirements. 
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Overview  of  DRRS 
Program  Management  and 
Oversight  Structure 


Since  its  establishment,  the  DIO  has  operated  within  the  Office  of  USD 
(P&R)  and  has  relied  on  multiple  contractors.  To  provide  governance  of 
DRRS,  and  enhance  communication  between  the  development  community, 
represented  by  the  DIO  and  contractors,  and  the  user  community,  which 
includes  the  Joint  Staff,  military  services,  and  combatant  commands,  USD 
(P&R)  established  various  bodies  with  representatives  from  the  user 
community,  including  military  services,  combatant  commands,  and  the 
defense  agencies.  Representatives  from  the  Office  of  USD  (P&R)  and  the 
Joint  Staff  currently  serve  as  cochairs  of  the  various  bodies.  DRRS  Battle 
Staffs  comprise  colonels,  Navy  captains,  and  similar-graded  civilians.  They 
track  DRRS  development  and  identify  issues  with  the  system.  At  the  one- 
star  level,  the  DRRS  General  and  Flag  Officer  Steering  Committee 
discusses  issues  raised  by  the  Battle  Staff.  In  December  2007,  USD  (P&R) 
created  a  committee  at  the  three-star  level,  referred  to  as  the  DRRS 
Executive  Committee.  Its  charter,  finalized  about  a  year  later  in  January 
2009,  calls  for  the  committee  to  review  and  approve  proposals  and  plans  to 
establish  policy,  processes,  and  system  requirements  for  DRRS,  including 
approving  software  development  milestones  required  to  reach  objectives. 

To  ensure  that  leadership  is  provided  for  the  direction,  oversight,  and 
execution  of  DOD’s  business  transformation  efforts,  including  business 
systems  modernization  efforts  such  as  DRRS,  DOD  relies  on  several 
entities.  These  entities  include  the  Defense  Business  Systems  Management 
Committee,  which  is  chaired  by  the  Deputy  Secretary  of  Defense  and 
serves  as  the  department’s  highest-ranking  governance  body  and  the 
approval  authority  for  business  systems  modernization  activities;  the 
Investment  Review  Boards,  which  are  chartered  by  the  Principal  Staff 
Assistants — senior  leaders  from  various  offices  within  DOD — and  serve  as 
the  review  and  certification  bodies  for  business  system  investments  in 
their  respective  areas  of  responsibility;11  and  the  Business  Transformation 
Agency,  which  is  responsible  for  supporting  the  Investment  Review 
Boards  and  for  leading  and  coordinating  business  transformation  efforts 
across  the  department.  Among  other  things,  the  Business  Transformation 
Agency  supports  the  Office  of  the  Under  Secretary  of  Defense,  Acquisition, 


uThe  four  Investment  Review  Boards  are  for  (1)  Financial  Management,  established  by  the 
Deputy  Under  Secretary  of  Defense  for  Financial  Management;  (2)  Weapon  Systems 
Lifecycle  Management  and  Materiel  Supply  and  Services  Management;  (3)  Real  Property 
and  Installations  Lifecycle  Management,  both  established  by  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics;  and  (4)  Human  Resources  Management, 
established  by  USD  (P&R). 
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Technology  and  Logistics  in  conducting  system  acquisition  risk 
assessments.12 


Disciplined  System 
Acquisition  and  Oversight 
Are  Keys  to  Program 
Success 


Our  research  and  evaluations  of  information  technology  programs, 
including  business  systems  modernization  efforts  within  DOD,  have  shown 
that  delivering  promised  system  capabilities  and  benefits  on  time  and 
within  budget  largely  depends  on  the  extent  to  which  key  program 
management  disciplines  are  employed  by  an  adequately  staffed  program 
management  office.  Among  other  things,  these  disciplines  include  a 
number  of  practices  associated  with  effectively  developing  and  managing 
system  requirements,  adequately  testing  system  capabilities,  and  reliably 
scheduling  the  work  to  be  performed.  They  also  include  proactively 
managing  the  program  office’s  human  capital  needs,  and  promoting 
program  office  accountability  through  executive-level  program  oversight. 
DOD  acquisition  policies  and  guidance,  along  with  other  relevant 
guidance,  recognize  the  importance  of  these  management  and  oversight 
disciplines.13  As  we  have  previously  reported,  not  employing  these  and 
other  program  management  disciplines  increases  the  risk  that  system 
acquisitions  will  not  perform  as  intended  and  require  expensive  and  time- 
consuming  rework. 


12The  purpose  of  a  risk  assessment  is  to  reduce  systemic  risk  and  support  informed 
decision  making  by  focusing  on  delivering  business  capabilities  rapidly  and  at  a  reduced 
cost,  and  identifying  program  vulnerabilities  and  assisting  in  developing  mitigation 
solutions.  The  assessment  is  generally  performed  prior  to  major  acquisition  decisions, 
although  assessments  may  be  requested  for  other  reasons. 

13See  for  example,  Department  of  Defense  Instruction  5000.02,  Operation  of  the  Defense 
Acquisition  System  (Dec.  8,  2008);  Defense  Acquisition  University,  Test  and  Evaluation 
Management  Guide,  5th  ed.  (Fort  Belvoir,  Va.:  January  2005);  Institute  of  Electrical  and 
Electronics  Engineers,  Standard  1012-2004  for  Software  Verification  and  Validation 
(New  York,  N.Y.:  June  8,  2005);  GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best 
Practices  for  Developing  and  Managing  Capital  Program  Costs,  GAO-09-3SP 
(Washington,  D.C.:  March  2009);  and  GAO,  A  Model  of  Strategic  Human  Capital 
Management,  GAO-02-373SP  (Washington,  D.C.:  Mar.  15,  2002). 
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DOD  Disagreed  with 
GAO’s  Prior 
Recommendation  to 
Develop  an 
Implementation  Plan 
to  Guide  DRRS 
Development 


In  2003,  we  reported  that,  according  to  USD  (P&R)  officials,  DRRS  was  a 
large  endeavor,  and  that  development  would  be  challenging  and  require 
buy-in  from  many  users.14  We  also  reported  that  the  program  was  only  a 
concept  without  detailed  plans  to  guide  its  development  and 
implementation.  Based  on  the  status  of  the  program  at  that  time  and 
DOD’s  past  record  on  readiness  reporting  initiatives,  we  recommended 
that  the  Secretary  of  Defense  direct  the  Office  of  USD  (P&R)  to  develop  an 
implementation  plan  that  identified 

•  performance  goals  that  are  objective,  quantifiable,  and  measurable; 

•  performance  indicators  to  measure  outcomes; 


•  an  evaluation  plan  to  compare  program  results  with  established  goals; 
and 


•  milestones  to  guide  DRRS  development  to  the  planned  2007  full 
capability  date. 

DOD  did  not  agree  with  our  recommendation,  stating  that  it  had 
established  milestones,  cost  estimates,  functional  responsibilities, 
expected  outcomes,  and  detailed  plans  for  specific  information  technology 
requirements  and  progress  indicators.  In  evaluating  the  DOD  comments, 
we  noted  that  DOD  had  established  only  two  milestones — initial  capability 
in  2004  and  full  capability  in  2007 — and  did  not  have  a  road  map  explaining 
the  steps  needed  to  achieve  full  capability  by  2007. 15 


DOD  Has  Not 
Effectively  Managed 
and  Overseen  the 
Acquisition  and 
Deployment  of  DRRS 


DOD  has  not  effectively  managed  and  overseen  the  acquisition  and 
deployment  of  DRRS  in  accordance  with  a  number  of  key  program 
management  disciplines  that  are  recognized  in  DOD  acquisition  policies 
and  guidance,  along  with  other  relevant  guidance,  and  are  fundamental  to 
delivering  a  system  that  performs  as  intended  on  time  and  within  budget. 
In  particular,  DRRS  requirements  have  not  been  effectively  developed  and 
managed,  and  DRRS  testing  has  not  been  adequately  performed  and 
managed.  Further,  DRRS  has  not  been  guided  by  a  reliable  schedule  of  the 


14The  users  of  readiness  data  include  the  Secretary  of  Defense,  the  military  services,  the 
Chairman  of  the  Joint  Chiefs  of  Staff,  the  combatant  commands,  defense  agencies,  and 
field  activities. 

15GAO,  Military  Readiness:  New  Reporting  System  Is  Inteyided  to  Address  Long-Standing 
Problems,  but  Better  Planning  Is  Needed,  GAO-03-456  (Washington,  D.C.:  Mar.  28,  2003). 
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work  needed  to  be  performed  and  the  key  activities  and  events  that  need 
to  occur.  These  program  management  weaknesses  can  be  attributed  in 
part  to  long-standing  limitations  in  program  office  staffing  and  oversight. 
As  a  result,  the  program  has  not  lived  up  to  the  requirements  set  for  it  by 
Congress,  and  the  department  has  not  received  value  from  the  program 
that  is  commensurate  with  the  time  and  money  invested — about  7  years 
and  $96.5  million.  Each  of  these  weaknesses  are  summarized  below  and 
discussed  in  detail  in  appendix  II. 


DRRS  Requirements  Have 
Not  Been  Effectively 
Developed  and  Managed 


According  to  DOD  and  other  relevant  guidance,  effective  requirements 
development  and  management  includes,  among  other  things,  (1) 
effectively  eliciting  user  needs  early  and  continuously  in  the  system  life- 
cycle  process,  (2)  establishing  a  stable  baseline  set  of  requirements  and 
placing  the  baseline  under  configuration  management,  (3)  ensuring  that 
system  requirements  are  traceable  backward  to  higher  level  business  or 
operational  requirements  (e.g.,  concept  of  operations)  and  forward  to 
system  design  documents  (e.g.,  software  requirements  specification)  and 
test  plans,  and  (4)  controlling  changes  to  baseline  requirements.  However, 
none  of  these  conditions  have  been  met  on  DRRS.  Specifically,  key  users 
have  only  recently  become  fully  engaged  in  developing  requirements,  and 
requirements  have  been  experiencing  considerable  change  and  are  not  yet 
stable.  Further,  different  levels  of  requirements  and  related  test  cases  have 
not  been  aligned  with  one  another,  and  changes  to  requirements  have  not 
been  effectively  controlled.  As  a  result,  efforts  to  develop  and  deliver 
initial  DRRS  capabilities  have  taken  longer  than  envisioned  and  these 
capabilities  have  not  lived  up  to  user  expectations.  These  failures  increase 
the  risk  of  future  DRRS  capabilities  not  meeting  expectations  and  increase 
the  likelihood  that  expensive  and  time-consuming  system  rework  will  be 
necessary. 


Recent  Actions  Have  Been 
Taken  to  Address  Limited  User 
Involvement  in  Developing  and 
Managing  Requirements 


Until  recently,  key  users  were  not  fully  or  effectively  engaged  in  DRRS 
requirements  development  and  management.  One  of  the  leading  practices 
associated  with  effective  requirements  development  is  engaging  system 
users  early  and  continuously  in  the  process  of  defining  requirements. 
However,  DIO  officials  and  representatives  from  the  military  services  and 
the  Joint  Staff  agree  that  until  recently,  key  users  were  not  effectively 
engaged  in  DRRS  requirements  development  and  management,  although 
they  disagree  at  to  why  user  involvement  has  suffered.  Regardless,  DRRS 
Executive  Committee  direction  has  improved  the  situation.  Specifically,  in 
January  2008,  the  committee  directed  the  Joint  Staff  to  conduct  an  analysis 
of  DRRS  capabilities,  referred  to  as  the  “capabilities  gap  analysis,”  which 
involved  the  entire  readiness  community  and  resulted  in  530  additional 
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DRRS  Requirements  Are  Not 
Stable 


Alignment  among 
Requirements  and  Related 
System  Design  and  Testing 
Artifacts  Has  Not  Been 
Demonstrated 


user  requirements.  In  our  view,  this  analysis  is  a  positive  step  in 
addressing  long-standing  limited  involvement  by  key  DRRS  users  in 
defining  requirements  that  has  contributed  to  significant  delays  in  the 
program,  as  discussed  later  in  the  report. 

As  of  April  2009,  DRRS  requirements  continued  to  be  in  a  state  of  flux. 
Establishing  an  authoritative  set  of  baseline  requirements  prior  to  system 
design  and  development  provides  a  stable  basis  for  designing,  developing, 
and  delivering  a  system  that  meets  its  users’  operational  needs.  However, 
the  fact  that  these  530  user  requirements  have  recently  been  identified 
means  that  the  suite  of  requirements  documentation  associated  with  the 
system,  such  as  the  detailed  system  requirements,  will  need  to  change  and 
thus  is  not  stable.  To  illustrate,  these  530  requirements  have  not  been  fully 
evaluated  by  the  DIO  and  the  DRRS  governance  boards  and  according  to 
program  officials,  have  not  yet  been  approved,  and  thus  their  impact  on 
the  program  is  not  clear.  Compounding  this  instability  in  the  DRRS 
requirements  is  the  fact  that  additional  changes  are  envisioned.  According 
to  program  officials,  the  changes  resulting  from  the  gap  analysis  and 
reflected  in  the  latest  version  of  the  DRRS  Concept  of  Operations,  which 
was  approved  by  the  DRRS  Executive  Committee  in  January  2009,  have 
yet  to  be  reflected  in  other  requirements  documents,  such  as  the  detailed 
system  requirements.  Although  defining  and  developing  requirements  is 
inherently  an  iterative  process,  having  a  baseline  set  of  requirements  that 
are  stable  is  a  prerequisite  to  effective  and  efficient  system  design  and 
development.  Without  them,  the  DIO  has  not  been  able  to  deliver  a  system 
that  meets  user  needs  on  time,  and  it  is  unlikely  that  future  development 
and  deployment  efforts  will  produce  better  results. 

During  our  review,  DIO  officials  could  not  demonstrate  that  requirements 
and  related  system  design  and  testing  artifacts  are  properly  aligned.  One  of 
the  leading  practices  associated  with  developing  and  managing 
requirements  is  maintaining  bidirectional  traceability  from  high-level 
operational  requirements  through  detailed  lower-level  requirements  and 
design  documents  to  test  cases.  We  attempted  on  three  separate  occasions 
to  verify  the  traceability  of  system  requirements  backwards  to  higher-level 
requirements  and  forward  to  lower-level  software  specifications  and  test 
cases,  and  each  time  we  found  that  traceability  did  not  exist.  DIO  and 
contractor  officials  attributed  the  absence  of  adequate  requirements 
traceability  to  the  ongoing  instability  in  requirements  and  efforts  to  update 
program  documentation.  Without  traceable  requirements,  the  DIO  does 
not  have  a  sufficient  basis  for  knowing  that  the  scope  of  the  design, 
development,  and  testing  efforts  will  produce  a  system  solution  on  time 
and  on  budget  and  that  will  meet  users’  operational  needs  and  perform  as 
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intended.  As  a  result,  the  risk  is  significant  that  expensive  and  time- 
consuming  system  rework  will  be  required. 

Changes  to  Requirements  Are  Since  the  inception  of  the  program  in  2002,  DRRS  has  been  developed  and 

Not  Being  Effectively  managed  without  a  formally  documented  and  approved  process  for 

Controlled  managing  changes  to  system  requirements.  Adopting  a  disciplined  process 

for  reviewing  and  accepting  changes  to  an  approved  baseline  set  of 
requirements  in  light  of  the  estimated  costs,  benefits,  and  risk  of  each 
proposed  change  is  a  recognized  best  practice.  However,  requirements 
management  and  change-control  plans  developed  in  2006  by  the  DRRS 
software  development  contractor,  according  to  DIO  officials,  were  not 
adequate.  To  address  this,  the  Joint  Staff  developed  what  it  referred  to  as  a 
conceptual  requirements  change-control  process  in  February  2008,  as  a 
basis  for  the  DIO  to  develop  more  detailed  plans  that  could  be 
implemented.  In  January  2009,  the  DIO  drafted  more  detailed  requirements 
management  and  configuration  management  plans,  the  latter  of  which  the 
DIO  updated  in  March  2009.  However,  the  plans  have  yet  to  be  approved 
and  implemented.  Until  the  DIO  effectively  controls  requirements  changes, 
it  increases  the  risk  of  needed  DRRS  capabilities  taking  longer  and  costing 
more  to  deliver  than  necessary. 


DRRS  Testing  Has  Not 
Been  Adequately 
Performed  and  Key  Test 
Management  Structures 
and  Controls  Have  Not 
Been  Established 


According  to  DOD  and  other  relevant  guidance,  system  testing  should  be 
progressive,  meaning  that  it  should  consist  of  a  series  of  test  events  that 
first  focus  on  the  performance  of  individual  system  components,  then  on 
the  performance  of  integrated  system  components,  followed  by  system- 
level  tests  that  focus  on  whether  the  system  (or  major  system  increments) 
are  acceptable,  interoperable  with  related  systems,  and  operationally 
suitable  to  users.  For  this  series  of  related  test  events  to  be  conducted 
effectively,  each  test  event  needs  to  be  executed  in  accordance  with  well- 
defined  test  plans,  the  results  of  each  test  event  need  to  be  captured  and 
used  to  ensure  that  problems  discovered  are  disclosed  and  corrected,  and 
all  test  events  need  to  be  governed  by  a  well-defined  test  management 
structure. 


However,  the  DIO  cannot  demonstrate  that  it  has  adequately  tested  any  of 
the  DRRS  increments,  referred  to  as  system  releases  and  subreleases,  even 
though  it  has  already  acquired  and  partially  deployed  a  subset  of  these 
increments.  Moreover,  the  DIO  has  yet  to  establish  the  test  management 
structures  and  controls  needed  to  effectively  execute  DRRS  testing  going 
forward.  More  specifically,  the  test  events  for  already  acquired,  as  well  as 
currently  deployed  and  operating,  DRRS  releases  and  subreleases  were 
not  based  on  well-defined  plans.  For  example,  the  test  plan  did  not  include 
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a  schedule  of  activities  to  be  performed  or  defined  roles  and 
responsibilities  for  performing  them.  Also,  the  test  plan  did  not 
consistently  include  test  entrance  and  exit  criteria,  a  test  defect 
management  process,  and  metrics  for  measuring  progress.  Further,  test 
events  have  not  been  fully  executed  in  accordance  with  plans,  or  executed 
in  a  verifiable  manner,  or  both.  For  example,  although  increments  of 
DRRS  functionality  have  been  put  into  production,  the  DIO  has  not 
performed  system  integration  testing,  system  acceptance  testing,  or 
operational  testing  on  any  DRRS  release  or  subrelease.  Moreover,  the 
results  of  all  executed  test  events  have  not  been  captured  and  used  to 
ensure  that  problems  discovered  were  disclosed  to  decision  makers,  and 
ultimately  corrected.  For  example,  the  DIO  has  not  captured  the  test 
results  for  at  least  20  out  of  63  DRRS  subreleases.  Test  results  that  were 
captured  did  not  include  key  elements,  such  as  entrance/exit  criteria 
status  and  unresolved  defects  and  applicable  resolution  plans. 

The  DIO  has  also  not  established  an  effective  test  management  structure 
to  include,  for  example,  a  clear  assignment  of  test  management  roles  and 
responsibilities,  or  a  reliable  schedule  of  planned  test  events. 
Compounding  this  absence  of  test  management  structures  and  controls  is 
the  fact  that  the  DIO  has  yet  to  define  how  the  development  and  testing  to 
date  of  a  series  of  system  increments  (system  releases  and  subreleases) 
relate  to  the  planned  development  and  testing  of  the  10  system  modules 
established  in  January  2009.  (See  table  1  for  a  list  and  description  of  these 
modules.)  Collectively,  this  means  that  it  is  unlikely,  that  already 
developed  and  deployed  DRRS  increments  can  perform  as  intended  and 
meet  user  operational  needs.  Equally  doubtful  are  the  chances  that  the 
DIO  can  adequately  ensure  that  yet-to-be  developed  DRRS  increments  will 
meet  expectations. 
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Table  1 :  DRRS  Capability  Modules 

Module 

Definition 

1. 

J  oint  Mission  Readiness 

Enables  selected  users  to  see  and  assess  the  highest 
strategic-level  "joint"  readiness  data. 

2. 

Unit  Mission  Readiness 

Captures  unit-level  readiness  data,  as  well  as 
authoritative  resources  data  (e.g.,  personnel, 
equipment)  fed  from  external  systems  owned  by  military 
services. 

3. 

Asset  Visibility 

Allows  authoritative  resources  data  from  military 
services  to  be  provided  in  near-real-time,  and  certifies 
them. 

4. 

Business  Intelligence 

P  rovides  the  ability  to  query  and  analyze  readiness  and 
asset  visibility  data,  based  on  the  desires  and  needs  of 
the  user. 

5. 

Installation  Readiness 

Allows  readiness  reporting  by  installations, 
training/firing  ranges  and  other  infrastructure  facilities. 

6. 

Readiness  Reviews 

Enables  forcewide  readiness  reviews  to  be  conducted, 
such  as  the  J  oint  Combat  Capability  Assessment 
review  process. 

7. 

Planning/Execution 

Support 

Allows  DRRS  to  interact  with  other  planning  systems 
and  processes,  such  as  the  Global  Force  Management 
and  Adaptive  P  lanning  and  Execution  communities. 

8. 

Historical  Data 

Focuses  on  the  historical  collection  and  presentation  of 
information.  Also  includes  the  Continuity  of  Operations 
(COOP)  capability. 

9. 

System  Architecture 

Meets  the  underlying  information  technology  system 
specifications,  such  as  standards  for  information 
assurance,  interoperability,  bandwidth,  and  other 
issues. 

10. 

End-User  Usability 

Fulfills  end-user  support  of  the  DRRS  application  to 
include  training,  customer  support,  and  documentation. 

Source 

:  DOD. 

DRRS  Has  Not  Been 
Guided  by  a  Reliable 
Schedule 


The  success  of  any  program  depends  in  part  on  having  a  reliable  schedule 
that  defines,  among  other  things,  when  work  activities  will  occur,  how 
long  they  will  take,  and  how  they  are  related  to  one  another.  From  its 
inception  in  2002  until  November  2008,  the  DIO  did  not  have  an  integrated 
master  schedule,  and  thus  has  long  been  allowed  to  proceed  without  a 
basis  for  executing  the  program  and  measuring  its  progress.  In  fact,  the 
only  milestone  that  we  could  identify  for  the  program  prior  to  November 
2008  was  the  date  that  DRRS  was  to  achieve  full  operational  capability, 
which  was  originally  estimated  to  occur  in  fiscal  year  2007,  but  later 
slipped  to  fiscal  year  2008  and  then  fiscal  year  2011,  and  is  now  fiscal  year 
2014-a  7-year  delay. 
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Moreover,  the  DRRS  integrated  master  schedule  that  was  first  developed 
in  November  2008,  and  was  updated  in  January  2009  and  again  in  April 
2009  to  address  limitations  that  we  identified  and  shared  with  the  program 
office,  is  still  not  reliable.  Specifically,  our  research  has  identified  nine 
practices  associated  with  developing  and  maintaining  a  reliable  schedule.16 
These  practices  are  (1)  capturing  all  key  activities,  (2)  sequencing  all  key 
activities,  (3)  assigning  resources  to  all  key  activities,  (4)  integrating  all 
key  activities  horizontally  and  vertically,  (5)  establishing  the  duration  of  all 
key  activities,  (6)  establishing  the  critical  path  for  all  key  activities,  (7) 
identifying  float  between  key  activities,  (8)  conducting  a  schedule  risk 
analysis,  and  (9)  updating  the  schedule  using  logic  and  durations  to 
determine  the  dates  for  all  key  activities. 17  The  program’s  latest  integrated 
master  schedule  does  not  address  three  of  the  practices  and  only  partially 
addresses  the  remaining  six.  For  example,  the  schedule  does  not  establish 
a  critical  path  for  all  key  activities,  nor  does  it  include  a  schedule  risk 
analysis,  and  it  is  not  being  updated  using  logic  and  durations  to  determine 
the  dates  for  all  key  activities.  In  addition,  the  schedule  introduces 
considerable  concurrency  across  key  activities  and  events  for  several 
modules,  which  introduces  increased  risk.  These  limitations  in  the 
program’s  latest  integrated  master  schedule,  coupled  with  the  program’s  7- 
year  slippage  to  date  and  continued  requirements  instability,  make  it  likely 
that  DRRS  will  incur  further  delays. 


DIO  Lacks  Adequate 
Staffing  and  an  Effective 
Approach  to  Meeting  Its 
Human  Capital  Needs 


The  DIO  does  not  currently  have  adequate  staff  to  fulfill  its  system 
acquisition  and  deployment  responsibilities,  and  it  has  not  managed  its 
staffing  needs  in  an  effective  manner.  Effective  human  capital 
management  should  include  an  assessment  of  the  core  competencies  and 
essential  knowledge,  skills,  and  abilities  needed  to  perform  key  program 
management  functions,  an  inventory  of  the  program’s  existing  workforce 
capabilities,  an  analysis  of  the  gap  between  the  assessed  needs  and  the 
existing  capabilities,  and  plans  for  filling  identified  gaps.  DIO  performs  a 
number  of  fundamental  DRRS  program  management  functions,  such  as 
acquisition  planning,  performance  management,  requirements 
development  and  management,  test  management,  contractor  tracking  and 
oversight,  quality  management,  and  configuration  management.  To 
effectively  perform  such  functions,  program  offices,  such  as  the  DIO,  need 


16GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 

17 Float  is  the  amount  of  time  an  activity  can  slip  before  affecting  the  critical  path. 
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to  have  not  only  well-defined  policies  and  procedures  and  support  tools 
for  each  of  these  functions,  but  also  sufficient  human  capital  to  implement 
the  processes  and  use  the  tools  throughout  the  program’s  life  cycle. 
However,  the  DIO  is  staffed  with  only  a  single  full-time  government 
employee — the  DIO  Director.  All  other  key  program  office  functions  are 
staffed  by  either  contractor  staff  or  staff  temporarily  detailed,  on  an  as- 
needed  basis,  from  other  DOD  organizations.  In  addition,  key  positions, 
such  as  performance  manager  and  test  manager,  have  either  not  been 
established  or  are  vacant.  According  to  DIO  and  contractor  officials,  they 
recognize  that  additional  program  management  staffing  is  needed  but 
stated  that  requests  for  additional  staff  had  not  been  approved  by  USD 
(P&R)  due  to  competing  demands  for  staffing.  Further,  they  stated  that  the 
requests  were  not  based  on  an  assessment  of  the  program’s  human  capital 
needs  and  the  gap  between  these  needs  and  its  onboard  workforce 
capabilities.  Until  DIO  adopts  a  strategic,  proactive  approach  to  managing 
its  human  capital  needs,  it  is  unlikely  that  it  will  have  an  adequate  basis  for 
obtaining  the  people  it  needs  to  effectively  and  efficiently  manage  DRRS. 


DOD  Executive  Oversight  A  key  principle  for  acquiring  and  deploying  system  investments  is  to 
of  DRRS  Has  Been  Limited  establish  a  senior-level  governance  body  to  oversee  the  investment  and 

hold  program  management  accountable  for  meeting  cost,  schedule,  and 
performance  commitments.  Moreover,  for  investments  that  are 
organization  wide  in  scope  and  introduce  new  ways  of  doing  business,  like 
DRRS,  the  membership  of  this  oversight  body  should  represent  all 
stakeholders  and  have  sufficient  organizational  seniority  to  commit  their 
respective  organizations  to  any  decisions  reached.  For  significant  system 
investments,  the  department’s  acquisition  process  provides  for  such 
executive  governance  bodies.  For  example,  Major  Automated  Information 
Systems,  which  are  investments  over  certain  dollar  thresholds  or  that  are 
designated  as  special  interest  because  of,  among  other  things,  their 
mission  importance,  are  reviewed  at  major  milestones  by  a  designated 
milestone  decision  authority. 18  These  authorities  are  supported  by  a  senior 
advisory  group,  known  as  the  Information  Technology  Acquisition  Board, 
which  comprises  senior  officials  from  the  Joint  Staff,  the  military 
departments,  and  staff  offices  within  the  Office  of  the  Secretary  of 


1SA  Major  Automated  Information  System  is  a  program  or  initiative  that  is  so  designated  by 
the  Assistant  Secretary  of  Defense  (Networks  and  Information  Integration)  /  Chief 
Information  Officer  or  that  is  estimated  to  require  program  costs  in  any  single  year  in 
excess  of  $32  million,  total  program  costs  in  excess  of  $126  million,  or  total  life-cycle  costs 
in  excess  of  $378  million  in  fiscal  year  2000  constant  dollars. 
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Defense.  In  addition,  all  business  system  investments  in  DOD  that  involve 
more  than  $1  million  in  obligations  are  subject  to  review  and  approval  by  a 
hierarchy  of  DOD  investment  review  boards  that  comprise  senior  DOD 
leaders,  including  the  Defense  Business  Systems  Management  Committee, 
which  is  chaired  by  the  Deputy  Secretary  of  Defense.  Through  these 
executive  oversight  bodies  and  their  associated  processes,  programs  are  to 
be,  among  other  things,  governed  according  to  corporate  needs  and 
priorities,  and  program  offices  are  to  be  held  accountable  for  meeting  cost, 
schedule,  and  performance  expectations. 

Until  April  2009,  DRRS  was  not  subject  to  any  of  DOD’s  established 
mechanisms  and  processes  for  overseeing  information  technology 
systems.  As  previously  discussed,  USD  (P&R)  established  the  DRRS  Battle 
Staff,  which  is  a  group  of  midlevel  military  officers  and  civilians  from 
DRRS  stakeholder  organizations,  and  it  established  a  higher-ranked 
General  and  Flag  Officer  Steering  Committee,  consisting  of  stakeholder 
representatives.  However,  neither  of  these  entities  had  specific  oversight 
responsibilities  or  decision-making  authority  for  DRRS.  Moreover,  neither 
was  responsible  for  holding  the  program  office  accountable  for  results. 
According  to  meeting  minutes  and  knowledgeable  officials,  these  entities 
met  on  an  irregular  basis  over  the  last  several  years,  with  as  much  as  a  1- 
year  gap  in  meeting  time  for  one  of  them,  to  discuss  DRRS  status  and 
related  issues. 

In  December  2007,  USD  (P&R)  recognized  the  need  for  a  more  senior-level 
and  formal  governance  body,  and  established  the  DRRS  Executive 
Committee.  Since  January  2008,  this  committee,  which  consists  of  top- 
level  representatives  from  stakeholder  organizations,  has  met  at  least 
seven  times.  In  January  2009,  the  DRRS  Executive  Committee’s  charter 
was  approved  by  the  Deputy  Under  Secretary  of  Defense  (Readiness)  and 
the  three-star  Director  of  the  Joint  Staff.  According  to  the  charter,  the 
committee  is  to  review  and  approve  proposals  and  plans  to  establish 
policy,  processes,  and  system  requirements  for  DRRS,  including  approving 
software  development  milestones  required  to  reach  objectives.  Consistent 
with  its  charter,  the  committee  has  thus  far  made  various  program-related 
decisions,  including  approving  a  DRRS  concept  of  operations  to  better 
inform  requirements  development,  and  directing  the  Joint  Staff  to  conduct 
an  analysis  to  identify  any  gaps  between  DRRS  requirements  and  user 
needs.  However,  the  committee  has  not  addressed  the  full  range  of 
acquisition  management  weaknesses  previously  discussed  in  this  report, 
and  it  has  not  taken  steps  to  ensure  that  the  program  office  is  accountable 
for  well-defined  program  baseline  requirements. 
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More  recently,  the  DOD  Human  Resources  Management  Investment 
Review  Board  and  the  Defense  Business  Systems  Management  Committee 
reviewed  DRRS  and  certified  and  approved,  respectively,  the  program  to 
invest  $24,625  million  in  fiscal  years  2009  and  2010.  These  entities 
comprise  senior  leadership  from  across  the  department,  including  the 
Deputy  Secretary  of  Defense  as  the  Defense  Business  Systems 
Management  Committee  Chair,  military  service  secretaries,  the  defense 
agency  heads,  principal  staff  assistants,  and  representatives  from  the  Joint 
Staff  and  combatant  commands.  However,  neither  the  Investment  Review 
Board’s  certification  nor  the  Defense  Business  Systems  Management 
Committee’s  approval  was  based  on  complete  and  accurate  information 
from  USD  (P&R).  Specifically,  the  certification  package  submitted  to  both 
oversight  bodies  by  the  USD  (P&R)  precertification  authority  (Office  of 
Readiness  Programming  and  Assessment)  stated  that  DRRS  was  on  track 
for  meeting  its  cost,  schedule,  and  performance  measures  and  highlighted 
no  program  risks  despite  the  weaknesses  discussed  in  this  report. 
According  to  the  chairwoman  of  the  Investment  Review  Board,  the  board 
does  not  have  a  process  or  the  resources  to  validate  the  information 
received  from  the  programs  that  it  reviews. 19  Moreover,  the  chairwoman 
stated  that  program  officials  did  not  make  the  board  aware  of  the  results 
of  our  review  that  we  shared  with  the  DIO  prior  to  either  the  Investment 
Review  Board  or  Defense  Business  Systems  Management  Committee 
reviews.  Since  we  briefed  the  chairwoman,  the  Investment  Review  Board 
has  requested  that  the  DIO  provide  it  with  additional  information 
documenting  DRRS  compliance  with  applicable  DOD  regulations  and 
statutes. 

According  to  USD  (P&R)  and  DIO  officials,  DRRS  was  not  subject  to 
department  executive-level  oversight  for  almost  6  years  because,  among 
other  things,  they  did  not  consider  DRRS  to  be  a  more  complex 
information  technology  system.  Furthermore,  because  of  the  nature  of  the 
authority  provided  to  the  USD  (P&R)  in  the  DRRS  charter,  they  did  not 
believe  it  was  necessary  to  apply  the  same  type  of  oversight  to  DRRS  as 
other  information  systems  within  DOD.  This  absence  of  effective  oversight 
has  contributed  to  a  void  in  program  accountability  and  limited  prospects 
for  program  success. 


19 As  previously  noted,  the  Investment  Review  Board  is  responsible  for  all  business  system 
investments  in  DOD  that  involve  more  than  $1  million  in  obligations. 
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Some  DRRS  Features 
Are  Operational  but 
Are  Not  Fully 
Consistent  with 
Legislative 
Requirements  and 
DOD  Guidance 


DOD  has  implemented  DRRS  features  that  allow  users  to  report  certain 
mission  capabilities  that  were  not  reported  under  the  legacy  system,  but 
these  features  are  not  fully  consistent  with  legislative  requirements  and 
DOD  guidance;  and  DOD  has  not  yet  implemented  other  envisioned 
features  of  the  system.  While  some  users  are  consistently  reporting 
enhanced  capability  information,  reporting  from  other  users  has  been 
inconsistent.  In  addition,  DRRS  has  not  fully  addressed  the  challenges 
with  metrics  that  were  identified  prior  to  1999  when  Congress  required 
DOD  to  establish  a  new  readiness  reporting  system.  Users  have  also  noted 
that  DRRS  lacks  some  of  the  current  and  historical  data  and  connectivity 
with  DOD’s  planning  systems  necessary  to  manage  and  deploy  forces. 


DOD  Is  Providing 
Congress  with  DRRS 
Capability  Data  from  the 
Combatant  Commands, 
but  Services  Have  Not 
Consistently  Reported 
Capability  Data 


The  geographic  combatant  commands  are  capturing  enhanced  capability 
data  in  DRRS,  and  DOD’s  quarterly  readiness  reports  to  Congress 
currently  contain  this  information,  as  well  as  information  that  is  drawn 
from  DOD’s  legacy  readiness  reporting  system,  GSORTS.  However,  the 
military  services  have  not  consistently  used  the  enhanced  capability 
reporting  features  of  DRRS.  Because  DRRS  does  not  yet  fully  interface 
with  legacy  systems  to  allow  single  reporting  of  readiness  data,  the  Army 
and  Navy  developed  additional  system  interfaces  and  are  reporting  in 
DRRS.  Until  May  2009,  the  Marine  Corps  directed  its  units  to  report  only  in 
the  legacy  system  to  avoid  the  burden  of  dual  reporting.  The  Air  Force 
chose  not  to  develop  an  interface  and  instructed  its  units  to  report  in  both 
DRRS  and  the  legacy  system. 


DRRS  and  GSORTS  both  contain  capabilities  information  and  resource 
(numbers  of  personnel,  equipment  availability,  and  equipment  condition) 
and  training  data.  However,  DRRS  currently  provides  more  capabilities 
data  than  GSORTS.  When  Congress  directed  DOD  to  establish  a  new 
readiness  reporting  system,  GSORTS  was  already  providing  capability 
information  concerning  unit  capabilities  to  perform  the  missions  for  which 
they  were  organized  or  designed.20  More  recently,  some  of  the  military 
services  began  reporting  limited  capability  information  on  unit  capabilities 
to  perform  missions  other  than  those  that  they  were  organized  or  designed 


20The  primaiy  GSORTS  metric — the  “C”  rating — is  a  mission-focused  metric  that  is  based 
not  only  on  a  unit’s  resources  but  also  on  the  unit’s  capabilities  to  execute  training  to 
standards,  in  a  specified  environment. 
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to  perform  into  GSORTS.21  However,  DRRS  is  designed  to  capture 
capabilities  on  a  wider  variety  of  missions  and  mission  essential  tasks.  For 
example,  organizations  can  report  their  capabilities  to  conduct  missions 
associated  with  major  war  plans  and  operations  such  as  Operation  Iraqi 
Freedom  into  DRRS,  as  well  as  their  capabilities  to  perform  the  missions 
for  which  they  were  organized  or  designed.  DRRS  also  captures  capability 
information  from  a  wider  range  of  organizations  than  GSORTS.  Although 
the  primary  (monthly)  focus  is  on  operational  units  and  commands,  DRRS 
collects  and  displays  readiness  information  from  defense  agencies  and 
installations. 

Geographic  combatant  commands — such  as  U.S.  Central  Command,  U.S. 
Pacific  Command,  and  U.S.  Northern  Command — are  currently  reporting 
their  commands’  capabilities  to  execute  most  of  their  operations  and 
major  war  plans  in  DRRS.  DOD  reports  this  enhanced  capability 
information  from  the  geographic  combatant  commands  in  its  Quarterly 
Readiness  Report  to  Congress.  The  geographic  combatant  commands  are 
also  using  DRRS  to  report  their  capabilities  to  perform  headquarters-level, 
joint  mission  essential  tasks,  and  some  of  these  commands  utilize  DRRS  as 
their  primary  readiness  reporting  tool.  For  example,  U.S.  Northern 
Command  uses  DRRS  to  assess  risk  and  analyze  capability  gaps,  and  U.S. 
Pacific  Command  identifies  critical  shortfalls  by  evaluating  mission 
essential  task  assessments  that  are  captured  in  DRRS. 

While  DRRS  currently  has  the  necessary  software  to  collect  and  display 
these  enhanced  capability  data  from  organizations  at  all  levels  throughout 
DOD,  a  variety  of  technical  and  other  factors  have  hindered  service 
reporting  of  capability  data.22  As  a  result,  the  services  have  either 
developed  their  own  systems  to  report  required  readiness  data  or  have 
delayed  issuing  implementing  guidance  that  would  require  their  units  to 
report  standardized  mission  essential  task  data  in  DRRS.  By  2005,  DRRS 
was  able  to  collect  and  display  mission  essential  task  information  from 
any  organizations  that  had  access  to  a  Secure  Internet  Protocol  Router 


21The  GSORTS  metric  that  captures  information  concerning  a  unit’s  capability  to  execute 
these  other  assigned  or  directed  missions  is  the  percent  effective  metric.  This  percent 
effective  rating  is  a  subjective  assessment  that  is  based  on  a  commander’s  professional 
military  judgment.  It  is  based  on  a  number  of  considerations  but  not  strictly  tied  to 
resources  or  training  performance. 

22The  software  to  collect  and  display  this  enhanced  capability  data  has  been  in  place  for 
several  years  and  the  DIO  has  stated  that  DRRS  reached  its  initial  operating  capability 
when  the  system  was  able  to  collect  and  display  this  information. 
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Lack  of  Direct  Access  to  DRRS 


Delays  in  Developing  Software 
Tools  to  Input  Data 


Network  (SIPRNet)  workstation.23  In  August  2005,  USD  (P&R)  issued  a 
memorandum  that  directed  the  services  to  ensure  that  all  of  their 
GSORTS-reporting  units  were  reporting  mission  essential  task  capabilities 
in  DRRS  by  September  30,  2005. 24  The  memorandum  stated  that,  for 
tactical  units,  mission  essential  tasks  were  to  be  drawn  from  the  Service 
Universal  Task  List  and  standardized  across  like-type  entities,  such  as  tank 
battalions,  destroyers,  or  F-16  squadrons.26  However,  two  factors  that  have 
hindered  compliance  with  the  memorandum’s  direction  to  report  mission 
essential  task  capabilities  in  DRRS  are  discussed  below. 

While  DRRS  has  been  able  to  collect  and  display  mission  essential  task 
data  since  2005,  some  Army  and  Navy  users  did  not  have  the  means  to 
directly  access  DRRS  and  update  mission  essential  task  assessments.  For 
example,  some  ships  lacked  hardware  necessary  to  be  able  to  transmit 
their  mission  essential  task  data  directly  into  DRRS  while  at  sea.  In 
addition,  many  National  Guard  units  lacked,  and  still  lack,  direct  access  to 
the  SIPRNet  workstations  that  are  necessary  to  directly  input  mission 
essential  task  data  directly  into  DRRS.  However,  the  Army  and  the  Navy 
have  developed  systems,  respectively  designated  DRRS-A  and  DRRS-N 
that  interface  with  DRRS  and  thus  allow  all  of  their  units  to  report  mission 
essential  task  data.  After  Army  and  Navy  units  report  mission  essential 
task  data  in  their  respective  DRRS-A  and  DRRS-N  service  systems,  the 
services  transmit  these  data  to  DRRS.  As  a  result,  Army  and  Navy  officials 
told  us  that  they  are  currently  complying  with  the  requirement  to  ensure 
that  all  their  GSORTS-reporting  units  report  mission  essential  task  data  in 
DRRS. 

Unlike  the  Army  and  the  Navy,  the  Marine  Corps  and  the  Air  Force  have 
not  developed  their  own  systems  to  allow  their  units  to  use  a  single  tool  to 


23The  SIPRNet  is  operated  by  the  Defense  Information  Systems  Agency  and  is  DOD’s 
largest  interoperable  command  and  control  data  network.  In  addition  to  supporting  the 
input  of  data  to  DRRS,  the  SIPRNet  supports  the  Global  Command  and  Control  System,  the 
Defense  Message  System,  collaborative  planning,  and  numerous  other  classified  warfighter 
applications. 

24The  memorandum — Under  Secretary  of  Defense  for  Personnel  and  Readiness 
Memorandum,  Department  of  Defense  Readiness  Reporting  System  (DRRS)  Interim 
Implementation  Guidance,  (Aug.  10,  2005) — directed  units  report  mission  essential  task 
capabilities  in  DRRS  through  the  automated  ESORTS  reporting  tool. 

25When  missions  or  mission  essential  tasks  are  not  standardized,  capability  searches  may 
not  yield  desired  results  because  similar  units  are  measuring  their  capabilities  against 
different  task  conditions  and  standards. 
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enter  readiness  data  to  meet  Office  of  the  Secretary  of  Defense,  Chairman 
of  the  Joint  Chiefs  of  Staff,  and  service  readiness  reporting  requirements. 
While  the  DIO  has  developed  the  software  for  users  to  enter  mission 
essential  task  data  into  DRRS,  the  DIO  has  been  unsuccessful  in  attempts 
to  develop  a  tool  that  would  allow  Air  Force  and  Marine  Corps  users  to 
enter  readiness  data  to  meet  all  of  their  readiness  reporting  requirements 
through  DRRS.  As  a  result,  rather  than  reducing  the  burden  on  reporting 
units,  DRRS  has  actually  increased  the  burden  on  Air  Force  and  Marine 
Corps  units  because  they  are  now  required  to  report  readiness  information 
in  both  DRRS  and  GSORTS.  On  September  29,  2005,  USD  (P&R)  issued  a 
memorandum  stating  that  DRRS  is  the  single  readiness  reporting  system 
for  the  entire  Department  of  Defense  and  that  legacy  systems,  such  as 
GSORTS  and  associated  service  readiness  systems,  should  be  phased 
out.26  Since  that  time,  officials  have  discussed  whether  to  phase  out 
GSORTS  and  tentative  dates  for  this  action  have  slipped  several  times. 

In  2001,  the  Office  of  the  Deputy  Undersecretary  of  Defense  (Readiness) 
listed  reducing  reporting  burdens  as  a  key  characteristic  of  its  envisioned 
improved  readiness  assessment  system.  In  an  effort  to  eliminate  this 
burden  of  dual  reporting,  the  DIO  began  to  develop  a  “current  unit  status” 
tool  as  a  means  for  users  to  manage  unit-specific  readiness  data  and 
submit  required  reports  in  support  of  all  current  reporting  guidelines.27 
The  tool  was  to  minimize  the  burden  associated  with  dual  reporting  by 
collecting,  displaying,  and  integrating  resource  data  from  service 
authoritative  data  sources  with  GSORTS  and  DRRS.  However,  in 
December  2007,  the  DIO  reported  that  it  was  unable  to  deliver  the 
intended  functionality  of  the  “current  unit  status”  tool.  Instead,  the  DIO 
decided  to  develop  an  interim  reporting  tool,  known  as  the  SORTSREP 
tool,  which  would  not  provide  the  type  of  new  capabilities  envisioned  for 
the  “current  unit  status”  tool,  but  would  simply  replicate  the  functionality 
of  the  input  tool  that  the  Air  Force  and  Marines  already  used  to  input  data 
into  GSORTS.  After  delays,  and  10  months  of  effort,  the  DIO  delivered  the 
SORTSREP  tool  to  the  Marine  Corps  for  review.  Based  on  this  review,  in 
December,  2008,  the  Marine  Corps  provided  the  developers  and  the  DIO 
with  163  pages  of  detailed  descriptions  and  graphics  to  explain  the 
SORTSREP  tool’s  deficiencies.  It  then  informed  the  DIO  that  it  would  no 


~6Under  Secretary  of  Defense  for  Personnel  and  Readiness  Memorandum,  Status  of  Defense 
Readiness  Reporting  System  (DRRS)  Implementation  (Sept.  29,  2005). 

27Current  reporting  guidelines  require  the  services  to  report  both  in  the  Chairman  of  the 
Joint  Chiefs  of  Staffs  GSORTS  and  in  OSD’s  DRRS. 
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longer  expend  energy  and  resources  to  review  future  versions  of  the 
SORTSREP  tool  and  would  instead  look  at  leveraging  the  Army’s  or  Navy’s 
DRRS-A  or  DRRS-N  systems.  The  Air  Force  also  informed  the  DIO  that  it 
was  no  longer  interested  in  the  SORSTSREP  tool,  and  said  efforts  should 
be  focused  on  the  “current  unit  status”  tool  instead.  As  a  result,  the  Air 
Force  and  Marine  Corps  are  currently  faced  with  dual  reporting 
requirements,  as  illustrated  in  figure  1. 


Figure  1 :  Air  Force  and  Marine  Corps  Dual  Reporting  Requirements  to  Meet 
Readiness  Reporting  Guidelines 


RAS-IT:  Readiness  Assessment  System  Input  Tool 
Source:  GAO  based  on  Air  Force  and  Marine  Corps  information. 


On  March  3,  2009,  the  Air  Force  Deputy  Chief  of  Staff  (Operations,  Plans 
and  Requirements)  issued  a  memorandum  that  updated  the  Air  Force’s 
previous  implementing  guidance  and  directed  all  GSORTS-reporting  units 
to  begin  assessing  readiness  in  DRRS  based  on  standardized  core  task  lists 
within  90  days.28  As  a  result,  Air  Force  units  will  report  readiness  in  both 
DRRS  and  GSORTS  until  the  DIO  is  able  to  deliver  the  intended 
functionality  of  the  “current  unit  status”  tool. 

While  some  Marine  Corps  units  are  reporting  their  capabilities  in  DRRS, 
the  Marine  Corps  had  not  yet  directed  its  units  to  report  in  the  system  as 
of  May  2009.  The  Commandant  of  the  Marine  Corps  had  stated  that  he 


28Department  of  the  Air  Force  Memorandum,  Defense  Readiness  Reporting  System 
(DRRS)  Core  Unit  Mission  Essential  Task  List  ( METL )  Implementation  Guidance  (Mar. 
3,  2009). 
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supported  the  development  and  implementation  of  DRRS,  but  that  he 
would  not  direct  units  to  assess  mission  essential  tasks  in  DRRS  until  the 
system  met  its  stated  requirements  and  was  accepted  as  the  single 
readiness  reporting  system  of  record.  Marine  Corps  officials  said  that  they 
did  not  want  to  place  a  burden  on  operational  units,  which  were  fighting 
or  preparing  to  fight  a  war,  by  requiring  that  they  report  readiness  in  two 
different  systems.  After  we  completed  our  audit  work,  on  May  12,  2009,  the 
Marine  Corps  issued  an  administrative  message  that  required  that  units 
assess  their  mission  essential  tasks  and  missions  in  DRRS.  The  message 
stated  that  doing  so  would  improve  familiarity  with  DRRS,  which  will  lead 
to  an  easier  transition  when  the  Marine  Corps  fields  DRRS-Marine  Corps 
(DRRS-MC).29 

Without  a  viable  tool  for  inputting  data,  DRRS  is  not  fully  integrated  with 
GSORTS  or  with  the  service  readiness  reporting  systems  and  it  is  not 
capable  of  replacing  those  systems  since  it  does  not  capture  the  required 
data  that  are  contained  in  those  systems. 


DRRS  Enhanced 
Capability  Data  Are  Not 
Likely  to  Fully  Address  the 
Challenges  with  Metrics 
That  Existed  Prior  to  the 
Establishment  of  the 
System 


While  DRRS  is  being  used  to  provide  Congress  with  enhanced  capability 
information,  the  quality  of  DRRS  metrics  still  faces  the  same  challenges, 
including  limitations  in  timeliness,  precision,  and  objectivity  that  existed 
prior  to  1999  when  Congress  directed  DOD  to  establish  a  new  readiness 
reporting  system.  Section  117  of  Title  10  of  the  U.S.  Code  directed  the 
Secretary  of  Defense  to  establish  a  comprehensive  readiness  reporting 
system  to  measure  the  capability  of  the  armed  forces  in  an  “objective, 
accurate,  and  timely  manner.”  However,  the  enhanced  capability  data  that 
are  captured  in  DRRS  and  reported  to  Congress  are  no  more  timely  than 
the  readiness  data  that  were  being  provided  to  Congress  in  1999  using 
GSORTS.  Furthermore,  the  metrics  that  are  being  used  to  capture  the 
enhanced  capability  information  are  less  objective  and  precise  than  the 
metrics  that  were  used  to  report  readiness  in  1999. 


Timeliness  The  statute  directing  the  development  of  a  new  readiness  reporting  system 

requires  that  the  reporting  system  measure  in  a  timely  manner  the 
capability  of  the  armed  forces  to  carry  out  the  National  Security  Strategy, 
the  Secretary  of  Defense’s  defense  planning  guidance,  and  the  National 


"Marine  Corps  Administrative  Message  0307//09,  Update  to  Interim  Defense  Readiness 
Reporting  System  (DRRS)  Policy  and  Procedures  for  Marine  Corps  Units  and 
Installations  (May  12,  2009). 
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Precision  and  Objectivity 


Military  Strategy.  The  legislation  also  lists  a  number  of  specific 
requirements  related  to  frequency  of  measurements  and  updates.  For 
example,  the  law  requires  that  the  capability  of  units  to  conduct  their 
assigned  wartime  missions  be  measured  monthly,  and  that  units  report  any 
changes  in  their  overall  readiness  status  within  24  hours  of  an  event  that 
necessitated  the  change  in  readiness  status.30  In  its  DRRS  directive,  DOD 
assigned  USD  (P&R)  responsibility  for  ensuring  the  timeliness  of  DRRS 
information  and  data,  and  it  specified  that  DRRS  was  to  be  a  near-real-time 
readiness  reporting  system. 

While  DOD  is  reporting  readiness  information  to  Congress  on  a  quarterly 
basis  as  required,  and  units  are  measuring  readiness  on  a  monthly  basis, 
DRRS  is  not  a  near-real-time  reporting  system.  Specifically,  in  DRRS,  as  in 
GSORTS,  operational  commanders  assess  the  readiness  of  their 
organizations  on  a  monthly  basis  or  when  an  event  occurs  that  changes 
the  units’  overall  reported  readiness.  Thus,  DRRS  has  not  improved  the 
timeliness  of  the  key  readiness  data  that  are  reported  to  Congress. 
According  to  USD  (P&R)  officials,  DRRS  data  will  be  more  timely  than 
GSORTS  data  because  DRRS  will  update  underlying  data  from 
authoritative  data  sources  between  the  monthly  updates.31  However,  DRRS 
is  not  yet  capturing  all  the  data  from  the  authoritative  data  sources,  and 
according  to  service  officials,  the  service  systems  that  support  GSORTS 
also  draw  information  from  their  service  authoritative  data  sources 
between  the  monthly  updates.  Furthermore,  the  source  and  currency  of 
some  of  the  authoritative  data  that  are  currently  in  DRRS  are  not  clearly 
identified.  As  a  result,  some  users  told  us  that  they  are  reluctant  to  use 
DRRS  data  to  support  their  decisions. 

We  previously  reported  that  the  readiness  information  that  DOD  provided 
to  Congress  lacked  precision,  noting  that  GSORTS  readiness  measures 
that  differed  by  10  percentage  points  or  more  could  result  in  identical 
ratings,  with  DOD  often  not  reporting  the  detailed  information  behind  the 
ratings  outside  of  the  department.32  For  example,  units  that  were  at  90  and 


30The  law  also  lists  annual  requirements  for  training  establishments,  defense  installations, 
and  facilities  to  report  their  capabilities,  and  a  number  of  other  monthly,  quarterly,  and 
annual  requirements. 

31While  certain  data,  such  as  personnel  data,  are  captured  in  multiple  data  systems,  the 
services  or  OSD  designate  a  single  data  system  as  the  authoritative  database  for  each  type 
of  information. 

32GAO-03-456. 
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100  percent  of  their  authorized  personnel  strengths  both  were  reported  as 
P-1  in  DOD’s  reports  to  Congress.  In  2003,  USD  (P&R)  recognized  the 
imprecision  of  the  reported  metrics  from  GSORTS  and  noted  that  its 
efforts  to  develop  DRRS  would  allow  enhancements  to  reported  readiness 
data. 

As  previously  noted,  the  DRRS  capability  information  that  DOD  is 
reporting  to  Congress  covers  a  broader  range  of  missions  than  the 
GSORTS  information  that  was  provided  in  the  past.  However,  when 
comparing  the  DRRS  and  GSORTS  assessments  of  units’  capabilities  to 
perform  the  missions  for  which  the  units  were  organized  or  designed, 
DRRS  assessments  are  actually  less  precise  than  the  GSORTS 
assessments.  Specifically,  within  GSORTS,  overall  capability  assessments 
are  grouped  into  four  categories  based  on  four  percentage  ranges  for  the 
underlying  data.  For  example,  commanders  compare  on-hand  and  required 
levels  of  personnel  and  equipment.  Within  DRRS,  mission  essential  task 
assessments  are  reported  on  a  scale  that  includes  only  three  ratings — 
”yes”,  “no”,  and  “qualified  yes,”  which  can  include  any  assessments  that 
fall  between  the  two  extremes. 

The  law  directing  DOD  to  establish  a  new  readiness  reporting  system  also 
requires  that  the  system  measure  readiness  in  an  objective  manner. 
GSORTS  assessments  of  units’  capabilities  to  execute  the  missions  for 
which  they  were  organized  or  designed  are  based  on  objective  personnel 
and  equipment  data  and  training  information  that  may  include  both 
objective  and  subjective  measures.  Furthermore,  the  overall  capability 
assessment  in  GSORTS  is  based  on  an  objective  rule  that  calls  for  the 
overall  assessment  to  be  the  same  level  as  the  lowest  underlying  resource 
or  training  data  level.  For  example,  if  a  unit  reported  the  highest  personnel 
level  (P-1)  and  the  lowest  training  level  (T-4),  the  rules  in  the  GSORTS 
guidance  instruct  the  commander  to  rate  the  unit’s  overall  capability  at  the 
C-4  level.  Because  GSORTS  contains  these  objective  measures  and  rules,  it 
is  easy  to  evaluate  reported  readiness  to  see  if  it  aligns  with  established 
reporting  criteria.33 


33Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3401. 02A,  Global  Status  of  Resources 
and  Training  System  (GSORTS)  (Feb.  27,  2004),  permits  commanders  to  subjectively 
upgrade  or  downgrade  their  units’  overall  assessment,  although  not  their  individual 
resource  levels.  However,  these  changes  are  easy  to  identify  and  commanders  are  required 
to  provide  justifications  for  any  changes.  A  recent  GAO  review  of  Army  data  found  that 
commanders’  subjective  changes  constituted  less  than  10  percent  of  the  total  assessments. 


Page  27 


GAO-09-518  Military  Readiness 


Within  DRRS,  organizations  rate  their  capabilities  based  on  mission 
essential  tasks.  These  mission  essentials  tasks  have  conditions  and 
standards  associated  with  them.  The  conditions  specify  the  types  of 
environments  that  units  are  likely  to  face  as  they  execute  the  tasks,  such 
as  weather  conditions  and  political  or  cultural  factors.  Standards  describe 
what  it  means  for  the  unit  to  successfully  execute  the  task  under  specified 
conditions.  For  example,  a  unit  may  have  to  achieve  a  90  percent  success 
rate  for  measures  associated  with  the  task  being  assessed.  In  spite  of  these 
conditions  and  standards,  DRRS  mission  assessments  are  often  subjective 
rather  than  objective.  In  DRRS  program  guidance,  DOD  has  defined 
mission  essential  tasks  as  tasks  that  are  approved  by  the  commander  and 
that,  based  on  mission  analysis,  are  “absolutely  necessary,  indispensable, 
or  critical  to  mission  success.”  In  prior  briefings  and  reports  to  Congress, 
we  have  noted  examples  that  highlight  the  subjective  nature  of  DRRS 
mission  assessments.  For  example,  we  noted  that  one  commander  used 
his  professional  judgment  to  decide  that  his  command  was  “qualified”  to 
execute  a  mission  even  though  the  preponderance  of  the  “indispensable” 
tasks  that  supported  that  mission  were  rated  as  “no.”  In  contrast,  other 
commanders  used  their  professional  judgments  to  rate  missions  as 
“qualified”  based  on  one  or  more  “qualified”  tasks  among  many  “yes”  tasks. 


DRRS  Lacks  the  Complete 
Resource  and  Training 
Data  and  System 
Connectivity  Some  Users 
Need  to  Manage  and 
Deploy  Forces 


DRRS  does  not  have  all  of  the  resource,  training,  readiness  data,  and 
connectivity  with  the  department’s  operations  planning  and  execution 
system  that  the  services,  Joint  Staff,  and  certain  combatant  commands 
need  to  manage  and  deploy  forces.  As  a  result,  DRRS  is  not  yet  able  to 
operate  as  the  department’s  single  readiness  reporting  system,  as 
intended.  The  Secretary  of  Defense’s  and  the  Under  Secretary  of  Defense’s 
guidance  documents  recognize  that  DRRS  needs  to  support  the  data 
requirements  of  multiple  users.  For  example,  the  Secretary  of  Defense’s 
June  25,  2004,  memorandum  directed  USD  (P&R)  to  develop  DRRS  to 
support  the  data  requirements  identified  by  the  Chairman  of  the  Joint 
Chiefs  of  Staff,  the  combatant  commanders,  the  Secretaries  of  the  military 
departments,  and  the  Chief  of  the  National  Guard  Bureau.34  Furthermore, 
the  2002  DRRS  directive  noted  that  DRRS  was  to  build  upon  DOD’s 
existing  processes  and  readiness  assessment  tools  and  that  ESORTS 
information  (capability,  resource,  and  training),  where  appropriate,  is 


34The  memorandum’s  primary  purpose  was  to  provide  policy  implementation  establishing 
the  Commander  of  the  U.S.  Joint  Forces  Command  to  assume  the  duties  as  DOD’s  primary 
joint  force  provider. 
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Complete  and  Accurate  Current 
and  Historical  Data 


integrated  into  DOD’s  planning  systems  and  processes.  It  also  directed  the 
Chairman  of  the  Joint  Chiefs  of  Staff  to  maintain  the  GSORTS  database 
until  key  capabilities  of  DRRS  become  fully  operational. 

Officials  with  U.S.  Joint  Forces  Command  and  U.S.  Special  Operations 
Command  reported  that  historical  data  are  needed  to  manage  forces  and 
provide  users  the  ability  to  analyze  readiness  trends.35  Similarly,  service 
officials  stated  a  need  for  historical  data  so  they  can  manage  their  forces 
and  take  action  to  address  readiness  issues.  In  2005,  USD  (P&R)  reported 
that  unit  resource  data,  including  detailed  inventory  and  authorization 
data  on  personnel,  equipment,  supply,  and  ordnance  were  available  in 
DRRS.  However,  in  response  to  a  survey  we  conducted  in  December  2008, 
the  services  and  certain  combatant  commands  stated  that  necessary 
current  and  historical  resource  and  training  data  were  not  available  in 
DRRS.  For  example,  officials  from  all  four  services  responded  that  DRRS, 
at  that  time,  contained  less  than  half  of  their  GSORTS  resources  and 
training  data.  In  addition,  officials  from  U.S.  Joint  Forces  Command,  U.S. 
Special  Operations  Command,  the  U.S.  Strategic  Command,  and  the  U.S. 
Transportation  Command  all  responded  that  historical  resource  data  were 
not  available  in  DRRS.  We  confirmed  that  this  information  was  still  not 
available  when  we  concluded  our  review,  and  in  April,  2009,  the  DIO  said 
it  was  still  working  on  this  data  availability  issue. 

Furthermore,  user  organizations  have  reported  inaccuracies  in  the  data 
that  are  available  in  DRRS.  Marine  Corps  and  U.S.  Special  Operations 
Command  officials  stated  that  inconsistencies  between  DRRS  data  and  the 
data  in  other  readiness  systems  have  caused  them  to  adjudicate  the 
inconsistencies  by  contacting  their  subordinate  units  directly.  Army 
officials  noted  that  searches  of  DRRS  data  can  produce  different  results 
than  searches  in  the  Army’s  data  systems.  For  example,  they  noted  that  a 
DRRS  search  for  available  personnel  with  a  particular  occupational 
specialty  produced  erroneously  high  results  because  DRRS  did  not  employ 
the  appropriate  business  rules  when  conducting  the  search.  Specifically, 
DRRS  did  not  apply  a  business  rule  to  account  for  the  fact  that  an 
individual  person  can  possess  multiple  occupational  specialty  codes  but 
can  generally  fill  only  one  position  at  a  time.  DIO  officials  informed  us  that 
they  intend  to  correct  issues  with  the  accuracy  of  data  drawn  from 


35These  historical  data  would  include  training  data,  personnel  data,  and  equipment  supply 
and  condition  data  as  well  as  data  concerning  unit  capabilities  to  perform  the  missions  they 
were  organized  and  designed  to  perform. 
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external  databases.  However,  the  current  version  of  the  DRRS  Integrated 
Master  Schedule  indicates  that  the  ability  of  DRRS  to  provide  the 
capability  to  correctly  search,  manipulate,  and  display  current  and 
historical  GSORTS  and  mission  essential  task  data  will  not  be  complete 
until  June  2010.  As  a  result,  the  reliability  of  the  DRRS  data  is  likely  to 
remain  questionable  and  a  number  of  DOD  organizations  will  likely 
continue  to  rely  on  GSORTS  and  other  sources  of  readiness  data  to 
support  their  decision  making. 

Connections  to  Planning  One  important  DRRS  function  is  integration  with  DOD’s  planning  systems. 

Systems  Specifically,  the  2002  DRRS  directive  requires  USD  (P&R)  to  ensure  that, 

where  appropriate,  ESORTS  information  (capability,  resource,  and 
training)  is  compatible  and  integrated  into  DOD’s  planning  systems  and 
processes.  Global  force  management  is  one  of  the  DOD  planning 
processes  that  is  to  be  integrated  with  DRRS.  Global  Force  Management  is 
a  process  to  manage,  assess,  and  display  the  worldwide  disposition  of  U.S. 
forces,  providing  DOD  with  a  global  view  of  requirements  and  availability 
of  forces  to  meet  those  requirements.  The  integration  of  DRRS  with  global 
force  management  planning  processes  is  supposed  to  allow  DOD  to  link 
force  structure,  resources,  and  capabilities  data  to  support  analyses,  and 
thus  help  global  force  managers  fill  requests  for  forces  or  capabilities. 

Officials  from  the  four  organizations  with  primary  responsibilities  for 
providing  forces  (U.S.  Joint  Forces  Command,  U.S.  Special  Operations 
Command,  U.S.  Strategic  Command,  and  U.S.  Transportation  Command) 
all  stated  that  they  are  unable  to  effectively  use  DRRS  to  search  for  units 
that  will  meet  requested  capabilities.  These  commands  also  reported  that 
DRRS  does  not  currently  contain  the  information  and  tools  necessary  to 
support  global  force  management.  For  example,  officials  from  U.S. 
Northern  Command  told  us  that  when  they  used  DRRS  to  search  for 
available  helicopters  of  a  certain  type,  they  found  thousands,  but  when 
U.S.  Joint  Forces  Command  did  the  same  DRRS  search  they  found 
hundreds.  The  current  version  of  the  DRRS  Integrated  Master  Schedule 
indicates  that  DRRS  will  not  be  able  to  fully  support  global  force 
management  until  March  2011.  As  a  result,  these  commands  continue  to 
rely  on  GSORTS  rather  than  DRRS  to  support  their  planning  and  sourcing 
decisions. 


Conclusions 


DRRS  is  not  currently  and  consistently  providing  timely,  objective,  and 
accurate  information,  and  it  is  not  exactly  clear  where  the  department 
stands  in  its  efforts  to  meet  this  expectation  because  system  requirements 
remain  in  a  state  of  flux,  and  the  program  office  lacks  disciplined  program 


Page  30 


GAO-09-518  Military  Readiness 


management  and  results  information  due  to  a  long-standing  lack  of  rigor  in 
its  approach  to  acquiring  and  deploying  system  capabilities.  This  situation 
can  be  attributed,  in  part,  to  long-standing  limitations  in  the  program 
office’s  focus  on  acquiring  human  capital  skills  needed  to  manage  such  a 
complex  initiative.  It  can  also  be  linked  to  many  of  years  of  limited 
program  office  oversight  and  accountability.  Although  program  oversight 
has  recently  increased,  oversight  bodies  have  not  had  sufficient  visibility 
into  the  program’s  many  management  weaknesses. 

DRRS  is  providing  Congress  and  readiness  users  with  additional  mission 
and  mission  essential  task  capability  data  that  were  not  available  in 
GSORTS.  However,  after  investing  about  7  years  and  about  $96.5  million  in 
developing  and  implementing  DRRS,  the  system’s  schedule  has  been 
extended,  requirements  are  not  stable,  and  the  system  still  does  not  meet 
congressional  and  DOD  requirements  for  a  comprehensive  readiness 
reporting  system  to  assess  readiness  and  help  decision  makers  manage 
forces  needed  to  conduct  combat  and  contingency  operations  around  the 
world.  Given  DRRS  performance  and  management  weaknesses,  it  is 
critical  that  immediate  action  be  taken  to  put  the  program  on  track  and 
position  it  for  success.  Without  this  action,  it  is  likely  that  DRRS  will  cost 
more  to  develop  and  deploy  than  necessary  and  that  DOD  will  not  have  a 
comprehensive  reporting  system  that  meets  the  needs  of  all  the  decision 
makers  who  rely  on  accurate,  timely,  and  complete  readiness  information. 


Recommendations  for  af^rcss  the  risks  facing  DOD  in  its  acquisition  and  deployment  of 

DRRS,  and  to  increase  the  chances  of  DRRS  meeting  the  needs  of  the  DOD 
Executive  Action  readiness  community  and  Congress,  we  recommend  that  the  Secretary  of 

Defense  direct  the  Deputy  Secretary  of  Defense,  as  the  Chair  of  the 
Defense  Business  Systems  Management  Committee,  to  reconsider  the 
committee’s  recent  approval  of  DRRS  planned  investment  for  fiscal  years 
2009  and  2010,  and  convene  the  Defense  Business  Systems  Management 
Committee  to  review  the  program’s  past  performance  and  the  DIO’s 
capability  to  manage  and  deliver  DRRS  going  forward. 

To  fully  inform  this  Defense  Business  Systems  Management  Committee 
review,  we  also  recommend  that  the  Secretary  direct  the  Deputy  Secretary 
to  have  the  Director  of  the  Business  Transformation  Agency,  using  the 
appropriate  team  of  functional  and  technical  experts  and  the  established 
risk  assessment  methodology,  conduct  a  program  risk  assessment  of 
DRRS,  and  to  use  the  findings  in  our  report  and  the  risk  assessment  to 
decide  how  to  redirect  the  program’s  structure,  approach,  funding, 
management,  and  oversight.  In  this  regard,  we  recommend  that  the 
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Secretary  direct  the  Deputy  Secretary  to  solicit  the  advice  and 
recommendations  of  the  DRRS  Executive  Committee. 

We  also  recommend  that  the  Secretary,  through  the  appropriate  chain  of 
command,  take  steps  to  ensure  that  the  following  occur: 

1.  DRRS  requirements  are  effectively  developed  and  managed  with 
appropriate  input  from  the  services,  Joint  Staff,  and  combatant 
commanders,  including  (1)  establishing  an  authoritative  set  of  baseline 
requirements  prior  to  further  system  design  and  development;  (2) 
ensuring  that  the  different  levels  of  requirements  and  their  associated 
design  specifications  and  test  cases  are  aligned  with  one  another;  and 
(3)  developing  and  instituting  a  disciplined  process  for  reviewing  and 
accepting  changes  to  the  baseline  requirements  in  light  of  estimated 
costs,  benefits,  and  risk. 

2.  DRRS  testing  is  effectively  managed,  including  (1)  developing  test 
plans  and  procedures  for  each  system  increment  test  event  that 
include  a  schedule  of  planned  test  activities,  defined  roles  and 
responsibilities,  test  entrance  and  exit  criteria,  test  defect  management 
processes,  and  metrics  for  measuring  test  progress;  (2)  ensuring  that 
all  key  test  events  are  conducted  on  all  DRRS  increments;  (3) 
capturing,  analyzing,  reporting,  and  resolving  all  test  results  and  test 
defects  of  all  developed  and  tested  DRRS  increments;  and  (4) 
establishing  an  effective  test  management  structure  that  includes 
assigned  test  management  roles  and  responsibilities,  a  designated  test 
management  lead  and  a  supporting  working  group,  and  a  reliable 
schedule  of  test  events. 

3.  DRRS  integrated  master  schedule  is  reliable,  including  ensuring  that 
the  schedule  (1)  captures  all  activities  from  the  work  breakdown 
structure,  including  the  work  to  be  performed  and  the  resources  to  be 
used;  (2)  identifies  the  logical  sequencing  of  all  activities,  including 
defining  predecessor  and  successor  activities;  (3)  reflects  whether  all 
required  resources  will  be  available  when  needed  and  their  cost;  (4) 
ensures  that  all  activities  and  their  duration  are  not  summarized  at  a 
level  that  could  mask  critical  elements;  (5)  achieves  horizontal 
integration  in  the  schedule  by  ensuring  that  all  external  interfaces 
(hand-offs)  are  established  and  interdependencies  among  activities  are 
defined;  (6)  identifies  float  between  activities  by  ensuring  that  the 
linkages  among  all  activities  are  defined;  (7)  defines  a  critical  path  that 
runs  continuously  to  the  program’s  finish  date;  (8)  incorporates  the 
results  of  a  schedule  risk  analysis  to  determine  the  level  of  confidence 
in  meeting  the  program’s  activities  and  completion  date;  and  (9) 
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includes  the  actual  start  and  completion  dates  of  work  activities 
performed  so  that  the  impact  of  deviations  on  downstream  work  can 
be  proactively  addressed. 

4.  The  DRRS  program  office  is  staffed  on  the  basis  of  a  human  capital 
strategy  that  is  grounded  in  an  assessment  of  the  core  competencies 
and  essential  knowledge,  skills,  and  abilities  needed  to  perform  key 
DRRS  program  management  functions,  an  inventory  of  the  program 
office’s  existing  workforce  capabilities,  and  an  analysis  of  the  gap 
between  the  assessed  needs  and  the  existing  capabilities. 

5.  DRRS  is  developed  and  implemented  in  a  manner  that  does  not 
increase  the  reporting  burden  on  units  and  addresses  the  timeliness, 
precision,  and  objectivity  of  metrics  that  are  reported  to  Congress. 

To  ensure  that  these  and  other  DRRS  program  management  improvements 
and  activities  are  effectively  implemented  and  that  any  additional  funds  for 
DRRS  implementation  are  used  effectively  and  efficiently,  we  further 
recommend  that  the  Secretary  direct  the  Deputy  Secretary  to  ensure  that 
both  the  Human  Resources  Management  Investment  Review  Board  and 
the  DRRS  Executive  Committee  conduct  frequent  oversight  activities  of 
the  DRRS  program,  and  report  any  significant  issues  to  the  Deputy 
Secretary. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  signed  by  the  Deputy  Under 
Secretary  of  Defense  (Military  Personnel  Policy)  performing  the  duties  of 
the  Under  Secretary  of  Defense  (Personnel  and  Readiness),  DOD  stated 
that  the  report  is  flawed  in  its  assessment  of  DRRS,  noting  that  DRRS  is  a 
net-centric  application  that  provides  broad  and  detailed  visibility  on 
readiness  issues,  and  that  achieving  data  sharing  across  the  DOD 
enterprise  was  groundbreaking  work  fraught  with  barriers  and  obstacles, 
many  of  which  have  now  been  overcome.  In  addition,  DOD  stated  that  it 
was  disappointed  that  the  report  did  not  address  cultural  impediments 
that  it  considers  to  be  the  root  cause  of  many  of  the  issues  cited  in  the 
report  and  of  many  previous  congressional  concerns  on  readiness 
reporting.  DOD  further  stated  that  the  report  instead  focuses  on  past 
acquisition  process  and  software  development  problems  that  it  believes 
have  now  been  remedied  According  to  the  department,  this  focus,  coupled 
with  inaccurate  and  misleading  factual  information  included  in  the  report, 
led  us  to  develop  an  incomplete  picture  of  the  program.  Notwithstanding 
these  comments,  DOD  agreed  with  two  of  our  recommendations  and 
partially  agreed  with  a  third.  However,  it  disagreed  with  the  remaining  five 
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recommendations,  and  provided  comments  relative  to  each 
recommendation.  DOD’s  comments  are  reprinted  in  their  entirety  in 
appendix  III. 

In  summary,  we  do  not  agree  with  DOD’s  overall  characterization  of  our 
report  or  the  positions  it  has  taken  in  disagreeing  with  five  of  our 
recommendations,  finding  them  to  be  inconsistent  with  existing  guidance 
and  recognized  best  practices  on  system  acquisition  management, 
unsupported  by  verifiable  evidence,  and  in  conflict  with  the  facts  detailed 
in  our  report.  Further,  we  recognize  that  developing  DRRS  is  a  significant 
and  challenging  undertaking  that  involves  cultural  impediments.  As  a 
result,  our  report  explicitly  focuses  on  the  kind  of  program  management 
rigor  and  disciplines  needed  to  address  such  impediments  and 
successfully  acquire  complex  systems,  including  effective  requirements 
development  and  management  and  executive  oversight.  We  also  disagree 
that  our  report  focuses  on  past  issues  and  problems.  Rather,  it  provides 
evidence  that  demonstrates  a  long-standing  and  current  pattern  of  system 
acquisition  and  program  oversight  weaknesses  that  existed  when  we 
concluded  our  audit  work  and  that  DOD  has  not  provided  any  evidence  to 
demonstrate  has  been  corrected. 

In  addition,  we  would  emphasize  that  we  defined  our  objectives,  scope, 
and  methodology,  and  executed  our  audit  work  in  accordance  with 
generally  accepted  government  auditing  standards,  which  require  us  to 
subject  our  approach  as  well  as  the  results  of  our  audit  work  to  proven 
quality  assurance  checks  and  evidence  standards  that  require  us  to  seek 
documentation  rather  than  relying  solely  on  testimonial  evidence.  While 
we  support  any  departmental  efforts,  whether  completed  or  ongoing,  that 
would  address  the  significant  problems  cited  in  our  report,  we  note  that 
DOD,  in  its  comments,  did  not  specifically  cite  what  these  efforts  are  or 
provide  documentation  to  support  that  they  have  either  been  completed  or 
are  ongoing.  Therefore,  we  stand  by  our  findings  and  recommendations. 
Moreover,  we  are  concerned  that  in  light  of  the  program’s  significant  and 
long-standing  management  weaknesses,  the  department’s  decision  not  to 
pursue  recommendations  aimed  at  corrective  actions  for  five  of  our  eight 
recommendations  will  further  increase  risk  to  achieving  program  success, 
and  is  not  in  the  best  interests  of  the  military  readiness  community  or  the 
U.S.  taxpayer.  Accordingly,  we  encourage  the  department  to  reconsider  its 
position  when  it  submits  its  written  statement  of  the  actions  taken  on  our 
recommendations  to  the  Senate  Committee  on  Homeland  Security  and 
Governmental  Affairs  and  the  House  Committee  on  Oversight  and 
Government  Reform  ,  as  well  has  the  House  and  Senate  Committees  on 
Appropriations,  as  required  under  31  U.S.C.  720. 
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DOD’s  specific  comments  on  each  recommendation,  along  with  our 

responses  to  its  comments  follow. 

•  The  department  did  not  agree  with  our  recommendation  for  the  Deputy 
Secretary  of  Defense,  as  the  Chair  of  the  Defense  Business  Systems 
Management  Committee,  to  reconsider  the  committee’s  recent 
approval  of  DRRS  planned  investment  for  fiscal  years  2009  and  2010, 
and  to  convene  the  Defense  Business  Systems  Management  Committee 
to  review  the  program’s  past  performance  and  the  DIO’s  capability  to 
manage  and  deliver  DRRS  going  forward  in  deciding  how  best  to 
proceed.  In  this  regard,  DOD  stated  that  the  Investment  Review  Board 
certification  and  Defense  Business  Systems  Management  Committee 
approval  were  granted  in  compliance  with  the  established  processes.  It 
also  added  that  oversight  of  the  specific  issues  identified  in  this  report 
are  the  responsibility  of  the  DRRS  Executive  Committee,  which  it 
stated  has  and  will  continue  to  provide  appropriate  governance  for  this 
effort.  It  also  stated  that  USD  (P&R)  will  ensure  that  the  program  is 
compliant  with  all  acquisition  requirements  prior  to  submission  for 
further  certifications. 

We  do  not  question  whether  the  Investment  Review  Board  certification 
and  Defense  Business  Systems  Management  Committee  approval  were 
provided  in  accordance  with  established  processes,  as  this  is  not 
relevant  to  our  recommendation.  Rather,  our  point  is  that  the 
Investment  Review  Board  and  Defense  Business  Systems  Management 
Committee  were  provided,  and  thus  based  their  respective  decisions, 
on  erroneous  and  incomplete  information  about  DRRS  progress, 
management  weaknesses,  and  risks.  Moreover,  neither  the  Investment 
Review  Board  nor  the  Defense  Business  Systems  Management 
Committee  were  informed  about  the  findings  in  our  report,  even 
though  we  shared  each  of  them  with  the  DRRS  program  director  and 
other  DIO  officials  prior  to  both  the  Investment  Review  Board  and  the 
Defense  Business  Systems  Management  Committee  deliberations. 
Therefore,  while  the  Investment  Review  Board  certification  and  the 
Defense  Business  Systems  Management  Committee  approval  were 
granted  in  accordance  with  established  processes,  they  were  not  based 
on  a  full  disclosure  of  facts.  Moreover,  while  we  support  DOD’s 
comment  that  it  will  ensure  that  the  program  is  in  compliance  with  all 
acquisition  requirements  prior  to  further  certifications,  nothing 
precludes  the  board  or  the  committee  from  reconsidering  their 
respective  decisions  in  light  of  our  report.  With  respect  to  DOD’s 
comment  that  the  DRRS  Executive  Committee  has  and  will  continue  to 
provide  appropriate  governance  for  this  effort,  we  do  not  disagree  that 
the  DRRS  Executive  Committee  has  an  oversight  role.  However,  the 
DRRS  Executive  Committee  should  not  be  solely  responsible  for 
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oversight  of  the  specific  issues  in  our  report.  Both  the  Investment 
Review  Board  and  the  Defense  Business  Systems  Management 
Committee  provide  additional  layers  of  oversight  pursuant  to  law36  and 
DOD  policy.37  Accordingly,  we  stand  by  our  recommendation  as  it 
appropriately  seeks  to  have  the  Investment  Review  Board  and  Defense 
Business  Systems  Management  Committee,  in  collaboration  with  the 
DRRS  Executive  Committee,  act  in  a  manner  that  is  consistent  with 
their  respective  roles  as  defined  in  law.  In  doing  so,  our  intent  is  to 
promote  accountability  for  DRRS  progress  and  performance,  and 
prompt  action  to  address  the  many  risks  facing  the  program. 

•  The  department  agreed  with  our  recommendation  for  the  Deputy 
Secretary  of  Defense,  as  the  chair  of  the  Defense  Business  Systems 
Management  Committee,  to  have  the  Business  Transformation  Agency 
conduct  a  risk  assessment  of  DRRS,  and  with  the  advice  and 
recommendation  of  the  DRRS  Executive  Committee,  to  use  the  results 
of  this  assessment  and  the  findings  in  our  report  to  decide  how  to 
redirect  the  program.  In  this  regard,  the  department  stated  that  this 
assessment  will  be  complete  by  the  middle  of  fiscal  year  2010. 

•  The  department  did  not  agree  with  our  recommendation  for  ensuring 
that  DRRS  requirements  are  effectively  developed  and  managed.  In  this 
regard,  it  stated  that  the  program  has  an  authoritative  set  of  baseline 
requirements  established  with  an  effective  governance  process  for 
overseeing  the  requirements  management  process,  to  include  biweekly 
reviews  as  part  of  the  DRRS  configuration  control  process. 

We  do  not  agree.  At  the  time  we  concluded  our  work,  DRRS 
requirements  were  not  stable,  as  evidenced  by  the  fact  that  an 
additional  530  requirements  had  been  identified  that  the  DIO  was  still 
in  the  process  of  reviewing  and  had  yet  to  reach  a  position  on  their 
inclusion,  or  process  them  through  the  DRRS  change  control 
governance  process.  Moreover,  when  we  concluded  our  work,  this 
change  control  process  had  yet  to  be  approved  by  the  DRRS 
governance  structure.  As  we  state  in  our  report,  the  introduction  of 
such  a  large  number  of  requirements  provided  a  compelling  basis  for 
concluding  that  requirements  had  yet  to  progress  to  the  point  that  they 
could  be  considered  sufficiently  complete  and  correct  to  provide  a 


36Ronald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L.  No. 
108-375,  §  332  (2004)  (codified  at  10  U.S.C.  §§  186  and  2222). 

^Department  of  Defense  Instruction  Number  5000.02,  Operation  of  the  Defense 
Acquisition  System,  Enclosure  11  (December  8,  2008). 
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stable  baseline.  Our  recommendation  also  noted  that  the  Secretary 
should  take  steps  to  ensure  that  the  different  levels  of  requirements  be 
aligned  with  one  another.  DOD’s  comments  did  not  address  this  aspect 
of  our  recommendation. 

•  The  department  did  not  agree  with  our  recommendation  for  ensuring 
that  DRRS  testing  is  effectively  managed.  In  this  regard,  it  stated  that 
DRRS  testing  is  already  in  place  and  performing  effectively,  and  stated, 
among  other  things,  that  (1)  the  DIO  goes  through  a  rigorous  testing 
regimen  that  includes  documenting  test  plans  with  user  test  cases  for 
each  incremental  release  to  include  utilizing  system  integration, 
acceptance,  interoperability,  and  operational  testing;  (2)  user  test  cases 
and  functionality  are  validated  by  designated  testers  independent  of  the 
developers  prior  to  a  deployment;  and  (3)  for  interoperability  testing 
the  DIO  has  a  designated  test  director  and  the  Joint  Interoperability 
Test  Command  (JITC)  is  the  designated  interoperability  and 
operational  test  activity. 

We  do  not  agree.  As  our  report  concludes,  DRRS  testing  has  not  been 
effectively  managed  because  it  has  not  followed  a  rigorous  testing 
regimen  that  includes  documented  test  plans,  cases,  and  procedures. 

To  support  this  conclusion,  our  report  cites  numerous  examples  of  test 
planning  and  execution  weaknesses,  as  well  as  the  DIO’s  repeated 
inability  to  demonstrate  through  requisite  documentation  that  the 
testing  performed  on  DRRS  has  been  adequate.  Our  report  shows  that 
test  events  for  already  acquired,  as  well  as  currently  deployed  and 
operating,  DRRS  releases  and  subreleases  were  not  based  on  well- 
defined  plans  and  DOD  had  not  filled  its  testing  director  vacancy. 
Further,  our  report  shows  that  test  events  were  not  fully  executed  in 
accordance  with  plans  that  did  exist,  or  executed  in  a  verifiable 
manner,  or  both.  For  example,  although  increments  of  DRRS 
functionality  had  been  put  into  production,  the  program  had  no 
documentation  (e.g.,  test  procedures,  test  cases,  test  results)  to  show 
that  the  program  office  had  performed  system  integration  testing, 
system  acceptance  testing,  or  operational  testing  on  any  DRRS  release 
or  subrelease,  even  though  the  DIO’s  test  strategy  stated  that  such  tests 
were  to  be  performed  before  system  capabilities  became  operational. 
Moreover,  evidence  showed  that  the  results  of  all  executed  test  events 
had  not  been  captured  and  used  to  ensure  that  problems  discovered 
were  disclosed  to  decision  makers,  and  ultimately  corrected.  With 
respect  to  DOD’s  comments  that  JITC  is  the  designated  lead  for 
interoperability  and  operational  testing,  our  report  recognizes  that  JITC 
is  to  conduct  both  interoperability  and  operational  testing  before  the 
system  is  deployed  and  put  into  production  (i.e.,  used  operationally). 
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However,  during  the  course  of  our  audit,  the  DIO  could  not  produce 
any  evidence  to  show  that  interoperability  and  operational  testing  of  all 
operating  system  increments  had  been  conducted. 

•  The  department  did  not  agree  with  our  recommendation  for  ensuring 
that  the  DRRS  integrated  master  schedule  is  reliable.  In  this  regard,  it 
stated  that  a  process  is  already  in  place  to  ensure  that  the  schedule  is 
current,  reliable,  and  meets  all  the  criteria  outlined  in  the 
recommendation. 

We  do  not  agree.  As  our  report  states,  an  integrated  master  schedule 
for  DRRS  did  not  exist  until  November  2008,  which  was  2  months  after 
we  first  requested  one.  Moreover,  following  our  feedback  to  the  DIO  on 
limitations  in  this  initial  version,  a  revised  integrated  master  schedule 
was  developed  in  January  2009,  which  was  also  not  reliable. 
Subsequently,  a  revised  integrated  master  schedule  was  developed  in 
April  2009.  However,  as  we  detail  in  our  report,  that  version  still 
contained  significant  weaknesses.  For  example,  it  did  not  establish  a 
critical  path  for  all  key  activities  or  include  a  schedule  risk  analysis, 
and  was  not  being  updated  using  logic  and  durations  to  determine  the 
dates  for  all  key  activities.  These  practices  are  fundamental  to 
producing  a  sufficiently  reliable  schedule  baseline  that  can  be  used  to 
measure  progress  and  forecast  slippages.  In  addition,  the  schedule 
introduced  considerable  concurrency  across  key  activities  and  events 
for  several  modules,  which  introduces  increased  risk.  Therefore,  we 
stand  by  our  recommendation. 

•  The  department  partially  agreed  with  our  recommendation  for  ensuring 
that  it  has  an  effective  human  capital  strategy.  In  this  regard,  it  stated 
that  actions  are  underway  to  add  more  full-time  civilian  support  to  the 
DIO,  and  that  plans  exist  to  convert  some  contractor  to  civilian  billets 
during  the  2010/2011  time  frame. 

We  support  the  department’s  actions  and  plans  described  in  its 
comments  to  address  the  DIO  human  capital  management  limitations 
discussed  in  our  report,  but  would  note  that  they  do  not  go  far  enough 
to  systematically  ensure  that  the  program  has  the  right  people  with  the 
right  skills  to  manage  the  program  in  both  the  near  term  and  the  long 
term.  To  accomplish  this,  the  department  needs  to  adopt  the  kind  of 
strategic  and  proactive  approach  to  DRRS  workforce  management  that 
our  report  describes  and  our  recommendation  embodies.  As  our 
evaluations  and  research  show,  failure  to  do  so  increases  the  risk  that 
the  program  office  will  not  have  the  people  it  needs  to  effectively  and 
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efficiently  manage  DRRS.  Therefore,  we  believe  that  the  department 
needs  to  fully  implement  our  recommendation. 

•  The  department  did  not  agree  with  our  recommendation  to  take  steps 
to  ensure  that  DRRS  is  developed  and  implemented  in  a  manner  that 
does  not  increase  the  reporting  burden  on  units  and  addresses  the 
timeliness,  precision,  and  objectivity  of  metrics  that  are  reported  to 
Congress.  In  this  regard,  it  stated  that  one  of  the  primary  tenets  of 
DRRS  has  been  to  reduce  reporting  requirements  on  the  war  fighter.  It 
also  stated  that  DRRS  is  already  using  state-of-the-art  technology  to 
ensure  that  near-real-time  data  are  available  for  the  war  fighters. 

Finally  it  stated  that  the  DRRS  governance  structure  that  is  currently  in 
place  ensures  that  DRRS  development  does  not  deviate  from  these  core 
principles. 

While  we  recognize  that  a  goal  of  DRRS  is  to  reduce  a  reporting  burden 
on  the  war  fighter,  we  disagree  with  the  department’s  position  because 
the  system  has  not  yet  achieved  this  goal.  As  our  report  states,  while 
the  DIO  has  developed  the  software  for  users  to  enter  mission  essential 
task  data  into  DRRS,  the  DIO  has  been  unsuccessful  in  attempts  to 
develop  a  tool  that  would  allow  Air  Force  and  Marine  Corps  users  to 
enter  readiness  data  to  meet  all  of  their  readiness  reporting 
requirements  through  DRRS.  As  a  result,  rather  than  reducing  the 
burden  on  reporting  units,  DRRS  actually  increased  the  burden  on  Air 
Force  and  Marine  Corps  units  because  they  were  required  to  report 
readiness  information  in  both  DRRS  and  GSORTS.  Without  a  viable 
tool  for  inputting  data,  DRRS  is  not  fully  integrated  with  GSORTS  or 
with  the  service  readiness  reporting  systems  and  it  is  not  capable  of 
replacing  those  systems  since  it  does  not  capture  the  required  data  that 
are  contained  in  those  systems.  In  addition,  the  DRRS  readiness  data 
that  are  currently  reported  to  Congress  are  not  near-real-time  data. 
Specifically,  the  periodicity  for  DRRS  capability  assessments  is  the 
same  as  the  legacy  GSORTS  system’s  readiness  reports — monthly  or 
when  an  event  occurs  that  changes  a  unit’s  overall  readiness. 
Furthermore,  our  report  shows  that  DRRS  mission  assessments  are 
often  subjective  and  imprecise  because  they  are  reported  on  a  scale 
that  includes  only  three  ratings — ”yes,”  “no,”  and  “qualified  yes,”  which 
can  include  any  assessments  that  fall  between  the  two  extremes. 
Therefore,  because  additional  actions  are  still  needed  to  reduce 
reporting  burdens  and  improve  the  timeliness,  precision,  and 
objectivity  of  the  DRRS  data  that  are  reported  to  Congress,  we  stand  by 
our  recommendation. 
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•  The  department  agreed  with  our  recommendation  for  ensuring  that 
both  the  Human  Resources  Management  Investment  Review  Board  and 
the  DRRS  Executive  Committee  conduct  frequent  oversight  activities 
of  the  DRRS  program  and  report  any  significant  issues  to  the  Deputy 
Secretary  of  Defense.  In  this  regard,  the  department  stated  that  the 
USD  (P&R)  component  acquisition  executive  is  working  with  the 
program  to  ensure  that  it  becomes  fully  compliant  with  all  acquisition 
requirements.  In  addition,  it  stated  that  the  acquisition  executive  will 
certify  to  the  Human  Resources  Investment  Review  Board  and  the 
Deputy  Chief  Management  Officer  of  compliance  prior  to  submission  of 
future  certification  requests.  Further,  it  stated  that  the  current  DRRS 
governance  process  will  provide  sustained  functional  oversight  of  the 
program  and  that  issues  that  arise  in  any  of  these  areas  will  be  elevated 
for  review,  as  appropriate.  We  believe  these  are  positive  steps. 


We  are  sending  copies  of  this  report  to  the  appropriate  congressional 
committees;  the  Secretary  of  Defense;  and  the  Director,  Office  of 
Management  and  Budget.  The  report  will  also  be  available  at  no  charge  on 
the  GAO  Web  site  at  http://www.gao.gov. 

If  you  or  your  staffs  have  questions  about  this  report,  please  contact  us  at 
pickups@gao.gov  or  hiter@gao.gov  or  at  our  respective  phone  numbers, 
(202)  512-9619  and  (202)  512-  3439.  Contact  points  for  our  Offices  of 
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Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  GAO  staff  who  made  key  contributions  to  this  report  are 
listed  in  appendix  IV. 


Sharon  L.  Pickup 

Director,  Defense  Capabilities  and  Management 


Randolph  C.  Hite 
Director,  Information 

Technology  Architecture  and  Systems  Issues 
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Our  objectives  were  to  assess  the  extent  to  which  (1)  the  Department  of 
Defense  (DOD)  has  effectively  managed  and  overseen  DRRS  acquisition 
and  deployment,  and  (2)  features  of  the  Defense  Readiness  Reporting 
System  (DRRS)  have  been  implemented  and  are  consistent  with  legislative 
requirements  and  DOD  guidance. 

We  did  not  evaluate  the  department’s  overall  ability  to  assess  the  readiness 
of  its  forces  or  the  extent  that  data  contained  in  any  of  its  readiness 
reporting  systems,  including  DRRS  and  the  Global  Status  of  Resources  and 
Training  System  (GSORTS),  reflect  capabilities,  deficiencies, 
vulnerabilities,  or  performance  issues.  Our  review  focused  on  acquisition 
and  program  management  issues,  such  as  requirements  management, 
schedule,  and  human  capital  requirements;  the  current  usage  of  DRRS;  and 
the  extent  to  which  DRRS’  features  address  legislative  requirements  and 
DOD  guidance. 

To  determine  the  extent  to  which  the  DRRS  acquisition  and  deployment 
has  been  effectively  managed  and  overseen,  we  focused  on  the  following 
acquisition  management  areas:  (1)  requirements  development  and 
management,  (2)  test  planning  and  execution,  (3)  DRRS  schedule 
reliability,  and  (4)  human  capital  planning.  In  doing  so,  we  analyzed  a 
range  of  program  documentation,  such  as  high-level  and  detailed-level 
requirements  documentation,  test  plans  and  reports,  the  current  DRRS 
schedule,  and  program  management  documentation  and  interviewed 
cognizant  program  and  contractor  officials. 

•  To  determine  the  extent  to  which  the  program  had  effectively 
implemented  requirements  development  and  management,  we 
reviewed  relevant  program  documentation,  such  as  the  concept  of 
operations  document,  capability  requirements  document,  software 
requirements  document,  requirements  traceability  matrix, 
configuration  management  plan,  and  the  program  management  plan,  as 
well  as  minutes  of  change  control  board  meetings,  and  evaluated  them 
against  relevant  guidance. 1  Moreover,  we  reviewed  briefing  slides  from 
meetings  of  DRRS  oversight  bodies  in  order  to  identify  concerns  about 


'The  Capability  Maturity  Model®  Integration  for  Development,  developed  by  the  Software 
Engineering  Institute  of  Carnegie  Mellon  University,  defines  key  practices  that  are 
recognized  hallmarks  for  successful  organizations  that,  if  effectively  implemented,  can 
greatly  increase  the  chances  of  successfully  developing  and  acquiring  software  and 
systems.  Carnegie  Mellon  Software  Engineering  Institute,  Capability  Maturity  Model® 
Integration  for  Development,  version  1.2  (Pittsburgh,  Pa.:  August  2006). 
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DRRS  expressed  by  representatives  from  the  DRRS  community  of 
users,  as  well  as  the  efforts  by  the  Joint  Staff  (at  the  direction  of  DRRS 
Executive  Committee)  to  identify  and  address  any  gaps  identified  by 
users  in  the  development  of  DRRS  requirements.  To  determine  the 
extent  to  which  the  program  has  maintained  traceability  backward  to 
high-level  business  operation  requirements  and  system  requirements, 
and  forward  to  system  design  specifications  and  test  plans,  we 
randomly  selected  60  program  requirements  and  traced  them  both 
backward  and  forward.  This  sample  was  designed  with  a  6  percent 
tolerable  error  rate  at  the  95  percent  level  of  confidence  so  that,  if  we 
found  zero  problems  in  our  sample,  we  could  conclude  statistically  that 
the  error  rate  was  less  than  5  percent.  In  addition,  we  interviewed 
program  and  development  contractor  officials  to  discuss  the 
requirements  development  and  management  process. 

•  To  determine  if  the  DRRS  Implementation  Office  (DIO)  is  effectively 
managing  DRRS  testing,  we  reviewed  relevant  documentation,  such  as 
the  DRRS  Test  and  Evaluation  Master  Plans  and  test  reports  and 
compared  them  to  DOD  and  other  relevant  guidance.2  Further,  we 
reviewed  developmental  test  plans  and  procedures  for  each 
release/subrelease  that  to  date  has  either  been  developed  or  fielded  and 
compared  them  with  best  practices  to  determine  whether  test  activities 
had  been  adequately  documented.  We  also  examined  test  results  and 
reports  for  the  already  acquired,  as  well  as  currently  deployed  and 
operating,  DRRS  releases  and  subreleases  and  compared  them  against 
plans  to  determine  whether  they  had  been  executed  in  accordance  with 
plans.  Moreover,  we  reviewed  key  test  documentation,  such  as  the 
Software  Version  Descriptions,  and  compared  them  against  relevant 
guidance  to  determine  whether  defect  data  were  being  captured, 
analyzed,  prioritized,  and  reported.  We  also  interviewed  program  and 
contractor  officials  to  gain  clarity  beyond  what  was  included  in  the 
program  documentation,  including  the  Defense  Information  Systems 
Agency’s  Joint  Interoperability  Test  Center  in  order  to  determine  the 
results  of  their  efforts  to  independently  test  DRRS  interoperability.  In 
addition,  to  determine  the  extent  to  which  the  program  had  effectively 
tested  its  system  requirements,  we  observed  the  DIO’s  efforts  to 


2See  for  example,  Department  of  Defense  Instruction  5000.02,  Operation  of  the  Defense 
Acquisition  System  (Dec.  8,  2008);  Defense  Acquisition  University,  Test  and  Evaluation 
Management  Guide,  5th  ed.  (Fort  Belvoir,  Va.:  January  2005);  Institute  of  Electrical  and 
Electronic  Engineers,  Standard  1012-2004 for  Software  Verification  and  Validation 
(New  York,  N.Y.:  June  8,  2005);  and  Software  Engineering  Institute,  Capability  Maturity 
Model  Integration  for  Acquisition  version  1.2  (Pittsburgh,  Pa.:  November  2007). 
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demonstrate  the  traceability  of  60  program  requirements  to  test  cases 
and  results.  This  sample  was  designed  with  a  5  percent  tolerable  error 
rate  at  the  95  percent  level  of  confidence  so  that,  if  we  found  zero 
problems  in  our  sample,  we  could  conclude  statistically  that  the  error 
rate  was  less  than  5  percent. 

•  To  determine  the  extent  to  which  the  program’s  schedule  reflects  key 
estimating  practices  that  are  fundamental  to  having  a  reliable  schedule, 
we  reviewed  the  DRRS  integrated  master  schedules  and  schedule 
estimates  and  compared  them  with  relevant  guidance.3  We  also  used 
schedule  analysis  software  tools  to  determine  whether  the  latest 
schedule  included  key  information,  such  as  the  activities  critical  to  on- 
time  completion  of  DRRS,  a  logical  sequence  of  activities,  and  evidence 
that  the  schedule  was  periodically  updated.  We  also  reviewed  the 
schedule  to  determine  the  time  frames  for  completing  key  program 
activities  and  to  determine  any  changes  to  key  milestones.  In  addition, 
we  shared  the  results  of  our  findings  with  program  and  contractor 
officials  and  asked  for  clarifications.  We  then  reviewed  the  revised 
schedule,  prepared  in  response  to  the  weaknesses  we  found,  and 
compared  it  with  relevant  guidance. 

•  To  evaluate  whether  DOD  is  adequately  providing  for  the  DRRS 
program’s  human  capital  needs,  we  compared  the  program’s  efforts 
against  relevant  criteria  and  guidance,  including  our  own  framework 
for  strategic  human  capital  management.4  In  doing  so,  we  reviewed  key 
program  documentation,  such  as  the  program  management  plan  and 
the  DIO  organizational  structure  to  determine  whether  it  reflected  key 
acquisition  functions  and  identified  whether  these  functions  were  being 
performed  by  government  or  contractor  officials.  We  interviewed  key 
officials  to  discuss  workforce  analysis  and  human  capital  planning 
efforts. 

•  To  determine  the  level  of  oversight  and  governance  available  to  the 
DRRS  community  of  users,  we  attended  relevant  meetings,  met  with 
officials  responsible  for  program  certification,  and  reviewed  relevant 
guidance  and  program  documentation.  Specifically,  we  attended  Battle 
Staff  meetings  and  analyzed  briefing  slides  and  meeting  minutes  from 


3GAO,  GAO  Cost  Estimating  and  Assessment.  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington  D.C.:  March  2009). 

4GAO,  A  Model  of  Strategic  Human  Capital  Management,  GAO-02-373SP  (Washington, 
D.C.:  Mar.  15,  2002). 
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the  DRRS  Executive  Committee,  General  and  Flag  Officer’s  Steering 
Committee,  and  Battle  Staff  meetings — the  main  DRRS  governance 
bodies.  In  addition,  we  reviewed  key  DRRS  certification  and  approval 
documentation  provided  by  the  Human  Resources  Management 
Investment  Review  Board,  such  as  economic  viability  analyses  and  the 
business  system  certification  dashboard  and  met  with  Investment 
Review  Board  officials  to  determine  the  basis  for  certifying  and 
approving  DRRS. 

To  determine  the  extent  to  which  the  features  of  DRRS  have  been 
implemented  and  are  consistent  with  legislative  requirements  and  DOD 
guidance,  we  first  examined  the  language  of  Section  117  of  Title  10  of  the 
United  States  Code,  which  directs  the  Secretary  of  Defense  to  establish  a 
comprehensive  readiness  reporting  system.  We  identified  criteria  for  this 
system  in  DOD’s  directive  formally  establishing  the  system.  5  We  evaluated 
the  system  by  conducting  interviews — see  table  2  below  for  a  list  of  these 
organizations — and  receiving  system  demonstrations  from  members  of  the 
readiness  community  to  determine  how  they  used  DRRS  and  how  their 
usage  compared  with  the  criteria  established  for  the  system.  We  also 
conducted  content  and  data  analysis  of  system  documents  and  briefing 
packages  provided  by  the  DIO  and  Joint  Staff.  In  order  to  capture  the 
broadest  amount  of  data  about  the  system  we  conducted  a  survey  of 
readiness  offices  at  all  of  the  service  headquarters,  combatant  commands, 
and  the  National  Guard  Bureau  regarding  how  DRRS  was  currently  being 
used  and  the  types  and  amount  of  data  available  in  the  system.6  In 
addition,  to  track  the  development  of  DRRS  capabilities,  we  attended 
Battle  Staff  meetings  and  analyzed  documentation  from  meetings  of  all  the 
DRRS  governance  bodies.  We  also  searched  for  and  extracted  information 
from  DRRS  in  order  to  support  other  GAO  ongoing  readiness  reviews. 
While  our  usage  of  the  system  was  not  intended  as  a  formal  test  of  the 
system,  our  general  observations  concerning  system  functionality  and  the 
range  of  available  data  were  consistent  with  the  observations  of  most 
other  users,  which  were  noted  in  our  survey. 


department  of  Defense  Directive  7730.65,  Department  of  Defense  Readiness  Reporting 
System  (DRRS)  (June  3,  2002). 

6The  reporting  officer  for  U.S.  Pacific  Command  indicated  that  he  felt  that  the  GAO’s 
survey  did  not  “accurately  and  fully  assess  the  usefulness  and  functionality  of  DRRS”  and 
provided  additional  written  information  to  support  his  survey  answers.  After  consultation 
with  GAO’s  office  of  Applied  Research  and  Methods  we  determined  the  survey  did  assess 
DRRS  functionality.  In  order  to  capture  U.S.  Pacific  Command’s  concerns,  we  incorporated 
the  information  it  provided  into  the  report  sections  describing  current  DRRS  usage. 
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Table  2:  Organizations  Interviewed  during  Our  Review 

Office  of  the  Under  Secretary  of  Defense  (Personnel  &  Readiness),  Arlington, 
Virginia 

U.S  Army 

Office  ofthe  Deputy  Chief  of  Staff  (Operations),  Arlington,  Virginia 
U.  S.  Army  Forces  Command,  Fort  McPherson,  Georgia 
U.S.  Army  Pacific,  FortShafter,  Flawaii 

Installation  ManagementCommand,  Pacific  Region  Office,  FortShafter,  Flawaii 
U.S.  Navy 

Fleadquarters  Navy,  Integrated  Fleet  Readiness,  Arlington,  Virginia 
Fleet  Forces  Command,  Norfolk,  Virginia 
U.S.  Pacific  Fleet,  Pearl  H arbor,  Flawaii 
U.S.  Air  Force 

Fleadquarters,  U.S.  Air  Force  (Readiness),  Arlington,  Virginia 
Air  Combat  Command,  Langley  Air  Force  Base,  Virginia 
U.S.  Pacific  Air  Forces,  Readiness  Branch,  Flickam  Air  Force  Base,  Flawaii 
13th  Air  Force,  Flickam  Air  Force  Base,  Flawaii 
U.S.  Marine  Corps 

Fleadquarters  Marine  Corps  (Readiness),  Arlington,  Virginia 
Marine  Forces  Command,  Norfolk,  Virginia 
Marine  Forces  Pacific,  Camp  Smith,  Flawaii 
Combat  Logistics  Battalion  3,  Kaneohe  Bay,  Flawaii 
U.S.  Central  Command,  MacDill  Air  Force  Base,  Florida 
U.S.  J  ointForces  Command,  Norfolk,  Virginia 
U.S.  Northern  Command,  Colorado  Springs,  Colorado 
U.S.  Pacific  Command,  Camp  Smith,  Flawaii 
U.S.  Special  Operations  Command,  MacDill  Air  Force  Base,  Florida 

Source:  GAO. 

We  conducted  our  work  from  April  2008  through  August  2009  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 
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Our  research  and  evaluations  of  information  technology  programs  have 
shown  that  the  ability  to  deliver  promised  system  capabilities  and  benefits 
on  time  and  within  budget  largely  depends  on  the  extent  to  which  key 
program  management  disciplines  are  employed  by  an  adequately  staffed 
program  management  office.  Among  other  things,  these  disciplines  include 
a  number  of  practices  associated  with  effectively  developing  and 
managing  system  requirements,  adequately  testing  system  capabilities,  and 
reliably  scheduling  the  work  to  be  performed.  They  also  include 
proactively  managing  the  program  office’s  human  capital  needs,  and 
promoting  program  office  accountability  through  effective  program 
oversight.  Department  of  Defense  (DOD)  acquisition  policies  and 
guidance,  along  with  other  relevant  guidance,  recognize  the  importance  of 
these  management  and  oversight  disciplines. 1  As  we  have  previously 
reported,  not  employing  these  and  other  program  management  disciplines 
increases  the  risk  that  system  acquisitions  will  not  perform  as  intended 
and  require  expensive  and  time-consuming  rework.  Defense  Readiness 
Reporting  System  (DRRS)  acquisition  and  deployment  has  for  years  not 
been  effectively  managed  in  accordance  with  these  key  program 
management  disciplines  that  are  recognized  in  DOD  and  other  relevant 
guidance,  and  are  fundamental  to  delivering  a  system  that  performs  as 
intended  on  time  and  within  budget. 


'See  for  example,  Department  of  Defense  Instruction  5000.02,  Operation  of  the  Defense 
Acquisition  System  (Dec.  8,  2008);  Defense  Acquisition  University,  Test  and  Evaluation 
Management  Guide,  5th  ed.  (Fort  Belvoir,  Va.:  January  2005),  Institute  of  Electrical  and 
Electronic  Engineers,  Standard  1012-2004  for  Software  Verification  and  Validation 
(New  York,  N.Y.:  June  8,  2005);  GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best 
Practices  for  Developing  and  Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington 
D.C.:  March  2009);  and  GAO,  A  Model  of  Strategic  Human  Capital  Management, 
GAO-02-373SP  (Washington,  D.C. :  Mar.'  15,  2002). 
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DRRS  Requirements 
Have  Not  Been 
Effectively  Developed 
and  Managed 


Well-defined  and  well-managed  requirements  are  a  cornerstone  of 
effective  system  development  and  acquisition.  According  to  recognized 
guidance,  documenting  and  implementing  a  disciplined  process  for 
developing  and  managing  requirements  can  help  to  reduce  the  risks  of 
producing  a  system  that  is  not  adequately  tested,  does  not  meet  user 
needs,  and  does  not  perform  as  intended.2  Effective  requirements 
development  and  management  includes,  among  other  things,  (1) 
effectively  eliciting  user  needs  early  and  continuously  in  the  system  life- 
cycle  process,  (2)  establishing  a  stable  baseline  set  of  requirements  and 
placing  this  baseline  under  configuration  management,  (3)  ensuring  that 
system  requirements  are  traceable  backward  to  higher  level  business  or 
operational  requirements  (e.g.,  concept  of  operations)  and  forward  to 
system  design  documents  (e.g.,  software  requirements  specification)  and 
test  plans,  and  (4)  controlling  changes  to  baseline  requirements. 


DRRS  requirements  have  not  been  effectively  developed  and  managed. 
Specifically,  (1)  key  users  have  only  recently  become  engaged  in 
developing  requirements,  (2)  requirements  have  been  experiencing 
considerable  change  and  are  not  yet  stable,  (3)  different  levels  of 
requirements  and  related  test  cases  have  not  been  aligned  with  one 
another,  and  (4)  changes  to  requirements  have  not  been  effectively 
controlled.  As  a  result,  efforts  to  develop  and  deliver  initial  DRRS 
capabilities  have  taken  longer  than  envisioned  and  these  capabilities  have 
not  lived  up  to  the  readiness  communities’  expectations.  These  failures 
increase  the  risk  of  future  DRRS  capabilities  not  meeting  expectations  and 
ensure  that  expensive  and  time-consuming  system  rework  will  be 
necessary. 


2The  Capability  Maturity  Model®  Integration  for  Development,  developed  by  the 
Software  Engineering  Institute  of  Carnegie  Mellon  University,  defines  key  practices  that 
are  recognized  hallmarks  for  successful  organizations  that,  if  effectively  implemented,  can 
greatly  increase  the  chances  of  successfully  developing  and  acquiring  software  and 
systems.  Carnegie  Mellon  Software  Engineering  Institute,  Capability  Maturity  Model® 
Integration  for  Development ,  version  1.2  (Pittsburgh,  Pa.:  August  2006). 
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Recent  Actions  Have  Been 
Taken  to  Address  Limited 
User  Involvement  in 
Developing  and  Managing 
Requirements 

To  the  DRRS  Implementation  Office’s  (DIO)  credit,  the  October  2008 
DRRS  Risk  Management  Plan  recognizes  this  by  stating  that  the  success  of 
DRRS  depends  on  participation  and  support  from  the  broad  readiness 
community,  which  includes  combatant  commands,  Joint  Staff,  and  the 
military  services.  However,  until  recently,  key  users  were  not  effectively 
engaged  in  DRRS  requirements  development  and  management,  although 
reasons  vary  why  they  have  not.  Specifically,  DIO  officials  told  us  that 
beginning  in  2002,  they  reached  out  to  all  user  groups — combatant 
commands,  Joint  Staff,  and  the  military  services — in  defining 
requirements.  For  example,  they  cited  a  July  2002  memorandum  issued  by 
the  Office  of  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness 
(USD  P&R)  that  encouraged  the  Director  of  the  Joint  Chiefs  of  Staff, 
Deputy  Commanders  of  the  Combat  Commands,  Service  Operations 
Deputies,  and  Directors  of  Defense  Agencies  to  actively  support  the  DRRS 
effort  by  ensuring  that  their  organizations  are  represented  at  Battle  Staff 
meetings.4  However,  these  officials  told  us  that  the  military  services  and 
Joint  Staff  chose  not  to  participate. 

In  contrast,  officials  from  these  user  groups  told  us  their  involvement  had 
been  limited  by  what  they  characterized  as  difficulties  in  submitting 
requirements  through  the  DRRS  governance  boards  that  were  in  place  at 
that  time.  For  example,  an  official  from  the  Joint  Forces  Command  said 
that  the  Forces  Battle  Staff  governance  board  did  not  meet  for  about  a 
year  between  2005  and  2006.  Further,  the  official  said  that  the  meetings 
that  were  held  did  not  offer  users  the  opportunity  to  discuss  their 
concerns  or  influence  the  requirements  process.  Similarly,  an  official  from 
the  Marine  Corps  cited  a  lack  of  clear  and  transparent  communication 
from  the  DIO  as  a  significant  impediment. 


One  of  the  leading  practices  associated  with  effective  requirements 
development  is  engaging  system  users  early  and  continuously  in  the 
process  of  defining  requirements.  As  we  have  previously  reported, 
assessing  user  needs  early  in  the  process  increases  the  probability  of 
success  in  defining,  designing,  and  delivering  a  system  that  meets  user 
needs  and  performs  as  intended.3 


3GAO,  Secure  Border  Initiative:  DHS  Needs  to  Address  Significant  Risks  in  Delivering 
Key  Technology  Investment,  GAO-08-1086  (Washington,  D.C.:  Sept.  22,  2008). 

4Office  of  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness  Memorandum, 
Implementing  the  Defense  Readiness  Reporting  System  (July  1,  2002). 
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Notwithstanding  this  lack  of  stakeholder  involvement  in  setting 
requirements,  the  Office  of  USD  (P&R)  developed  and  issued  a  DRRS 
concept  of  operations  in  2004,  which  DIO  officials  told  us  was  based  on 
input  from  the  combatant  commands,  relevant  departmental  directives,  6 
and  DRRS  governance  boards  (e.g.,  Battle  Staff).  In  our  view,  this 
document  provided  a  high-level  overview  of  proposed  DRRS  capabilities 
from  which  more  detailed  requirements  could  be  derived.  However,  the 
concept  of  operations  was  not  approved  by  all  key  players  in  the  readiness 
community.  Specifically,  DIO  officials  stated  that  the  document  had  not 
been  approved  by  the  military  services  and  the  Joint  Staff.  According  to 
these  officials,  the  reason  for  not  seeking  all  stakeholders’  approval,  and 
the  decision  to  begin  developing  more  detailed  requirements  in  the 
absence  of  an  approved  concept  of  operations,  was  that  the  2002  DRRS 
DOD  directive  provided  a  sufficient  basis  to  begin  developing  and 
deploying  what  they  anticipated  being  the  initial  versions  of  DRRS.6 

In  2008,  after  6  years  of  effort  to  define  DRRS  requirements  and  develop 
and  deploy  system  capabilities,  the  Joint  Staff,  at  the  direction  of  the 
DRRS  Executive  Committee,  conducted  an  analysis  of  DRRS 
capabilities — referred  to  as  the  “capabilities  gap  analysis.”  To  the  Joint 
Staffs  credit,  this  analysis  has  appropriately  focused  on  soliciting 
comments  from  the  entire  readiness  community  and  on  identifying  any 
gaps  between  the  DRRS  requirements  and  the  needs  of  this  community.  As 
will  be  discussed  in  the  next  section,  this  analysis  resulted  in  530 
additional  user  requirements. 

The  extended  period  of  limited  involvement  by  key  DRRS  users  in  defining 
a  concept  of  operations  and  related  capabilities  and  requirements  has 
impeded  efforts  to  develop  a  clear  understanding  of  DRRS  expectations, 
constraints,  and  limitations,  which,  in  turn,  has  contributed  to  significant 
delays  in  providing  the  readiness  community  with  needed  system  support. 
While  the  recent  Joint  Staff  action  to  engage  the  entire  DRRS  user 
community  is  a  positive  step  towards  overcoming  this  long-standing 
problem,  it  remains  to  be  seen  whether  this  engagement  will  produce 


department  of  Defense  Directive  7730.65,  Department  of  Defense  Readiness  Reporting 
System  (DRRS)  (June  3,  2002).  Additional  requirements  for  DRRS  have  been  established  in 
DOD  directives  and  instructions  that  govern  information  systems  including  information 
assurance,  net-centric  architecture,  net-centric  data  strategy,  and  information  system 
interoperability  and  supportability. 

department  of  Defense  Directive  7730.65,  Department  of  Defense  Readiness  Reporting 
System  (DRRS)  (June  3,  2002). 
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agreement  and  commitment  across  the  entire  readiness  user  community 
around  DRRS  requirements. 


DRRS  Requirements  Are  As  previously  noted,  establishing  an  authoritative  set  of  baseline 
Not  Stable  requirements  prior  to  system  design  and  development  is  necessary  to 

design,  develop,  and  deliver  a  system  that  performs  as  intended  and  meets 
users’  operational  needs. 1  In  general,  a  baselined  set  of  requirements  are 
those  that  are  defined  to  the  point  that  extensive  changes  are  not 
expected,  placed  under  configuration  management,  and  formally 
controlled.8 

DRRS  requirements  are  currently  in  a  state  of  flux.  Specifically,  the  fact 
that  530  new  user  requirements  have  recently  been  identified  means  that 
the  suite  of  requirements  documentation  associated  with  the  system  will 
need  to  be  changed  and  thus  are  not  stable.  To  illustrate,  program  officials 
told  us  that,  as  of  late  February  2009,  these  530  new  requirements  had  not 
been  fully  evaluated  by  the  DIO  and  DRRS  governance  boards  and  thus 
not  yet  approved.  As  a  result,  their  impact  on  the  program  is  not  clear. 

Compounding  this  instability  in  the  DRRS  requirements  is  the  fact  that 
additional  changes  are  envisioned.  According  to  program  officials,  the 
changes  resulting  from  the  gap  analysis  and  reflected  in  the  latest  version 
of  the  DRRS  concept  of  operations,  which  was  approved  by  the  DRRS 
Executive  Committee  in  January  2009,  have  yet  to  be  reflected  in  other 
requirements  documents,  such  as  the  detailed  system  requirements. 

Although  defining  and  developing  requirements  is  inherently  an  iterative 
process,  having  a  baseline  set  of  requirements  that  are  stable  is  a 
prerequisite  to  effective  and  efficient  development  of  an  operationally 
capable  and  suitable  system.  Without  them,  the  DIO  will  not  be  able  to 
deliver  a  system  that  meets  user  needs  on  time,  and  it  is  unlikely  that 
future  development  and  deployment  efforts  will  produce  better  results. 


7GAO-08-1086. 

sThe  Capability  Maturity  Model®  Integration  for  Development,  developed  by  the 
Software  Engineering  Institute  of  Carnegie  Mellon  University,  defines  key  practices  that 
are  recognized  hallmarks  for  successful  organizations  that,  if  effectively  implemented,  can 
greatly  increase  the  chances  of  successfully  developing  and  acquiring  software  and 
systems.  Carnegie  Mellon  Software  Engineering  Institute,  Capability  Maturity  Model® 
Integration  for  Development,  Version  1.2  (Pittsburgh,  Pa.:  August  2006). 
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Alignment  among 
Requirements  and  Related 
System  Design  and  Testing 
Artifacts  Has  Not  Been 
Demonstrated 


One  of  the  leading  practices  associated  with  developing  and  managing 
requirements  is  maintaining  bidirectional  traceability  from  high-level 
operational  requirements  (e.g.,  concept  of  operations  and  functional 
requirements)  through  detailed  lower-level  requirements  and  design 
documents  (e.g.,  software  requirements  specification)  to  test  cases.  Such 
traceability  is  often  accomplished  through  the  use  of  a  requirements 
traceability  matrix,  which  serves  as  a  crosswalk  between  different  levels 
of  related  requirements,  design,  and  testing  documentation.  The  DRRS 
program  management  plan  recognizes  the  importance  of  traceability, 
stating  that  requirements  are  to  be  documented  and  linked  to  acceptance 
tests,  scripts,  and  criteria. 


Despite  the  importance  of  traceability,  DIO  officials  could  not  demonstrate 
that  requirements  and  related  system  design  and  testing  artifacts  are 
properly  aligned.  Specifically,  we  attempted  on  three  separate  occasions 
to  verify  the  traceability  of  system  requirements  backward  to  higher-level 
requirements  and  forward  to  lower-level  software  specifications  and  test 
cases,  and  each  time  we  found  that  traceability  did  not  exist.  Each  attempt 
is  discussed  here: 


•  In  November  2008,  our  analysis  of  the  requirements  traceability  matrix 
and  the  software  requirements  specification  showed  significant 
inconsistencies.  For  example,  the  traceability  matrix  did  not  include  29 
requirements  that  were  included  in  the  software  requirements 
specification.  As  a  result,  we  did  not  have  an  authoritative  set  of 
requirements  to  use  to  generate  a  random  sample  of  requirements  to 
trace.  Program  officials  attributed  the  inconsistencies  to  delays  in 
updating  all  the  documents  to  reflect  the  aforementioned  capability  gap 
analysis.  They  also  stated  that  these  documents  would  be  updated  by 
December  2008. 

•  In  December  2008,  we  used  an  updated  requirements  traceability 
matrix  to  generate  a  randomized  sample  of  60  software  requirements 
specifications  and  observed  a  DIO  demonstration  of  the  traceability  of 
this  sample.  However,  DIO  officials  were  unable  to  demonstrate  for  us 
that  these  specifications  could  be  traced  backward  to  higher-level 
requirements  and  forward  to  test  cases.  Specifically,  attempts  to  trace 
the  first  21  requirements  forward  to  test  cases  failed,  and  DIO  officials 
stated  that  they  could  not  trace  the  60  requirements  backward  because 
the  associated  requirements  documents  were  still  being  updated. 
According  to  the  officials,  11  of  the  21  could  not  be  traced  forward 
because  these  were  implemented  prior  to  2006  and  the  related  test 
information  was  not  maintained  by  the  program  office  but  rather  was 
at  the  development  contractor’s  site.  They  added  that  the  remaining  10 
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either  lacked  test  case  information  or  test  results. 

•  In  February  2009,  we  used  an  updated  DIO-provided  requirements 
traceability  matrix,  a  capabilities  requirement  document,  and  software 
requirements  specification  to  generate  another  randomized  sample  of 
60  detailed  specifications.  We  then  observed  the  development 
contractor’s  demonstration  of  traceability  using  the  contractor’s 
requirements  management  tool.  Because  of  time  constraints,  this 
demonstration  focused  on  46  of  the  60  requirements,  and  it  showed 
that  adequate  traceability  still  did  not  exist.  Specifically,  38  of  the  46 
could  not  be  traced  backward  to  higher-level  requirements  or  forward 
to  test  cases.  This  means  that  about  83  percent  of  the  DRRS 
specifications  (96  percent  degree  of  confidence  of  being  between  72 
and  91  percent)  were  not  traceable.  Of  the  38,  14  did  not  trace  because 
of  incomplete  traceability  documentation;  6  due  to  inconsistent 
traceability  documentation;  3  due  to  requirements  not  being  resident  in 
the  tracking  tool;  and  16  due  to  no  actual  development  work  being 
started. 

In  addition,  none  of  the  46  requirements  were  traceable  to  the  January 
2009  concept  of  operations.  According  to  contractor  officials,  this  is 
because  the  newly  developed  capability  requirements  document  is 
considered  to  be  a  superset  of  the  concept  of  operations,  and  thus 
traceability  to  this  new  document  is  their  focus.  However,  they  were 
unable  to  demonstrate  traceability  to  the  requirements  in  either  the 
capability  requirements  document  or  the  concept  of  operations.  Further, 
we  also  found  numerous  inconsistencies  among  the  capabilities 
requirements  document,  software  requirements  specification,  and  the 
requirements  traceability  matrix.  For  example,  15  capabilities 
requirements  listed  on  the  traceability  matrix  were  not  listed  in  the 
capabilities  requirements  document,  but  were  listed  in  the  updated 
software  requirements  specification,  dated  February  2009.  Further,  one 
requirement  listed  in  the  traceability  matrix  was  not  listed  in  either  of 
these  documents.  One  possible  reason  for  these  inconsistencies  is  that  the 
traceability  matrix  was  prepared  manually,  rather  than  being  automatically 
generated  from  the  tool,  which  would  increase  the  probability  of  these  and 
other  discrepancies  caused  by  human  error.  Another  reason  cited  by 
program  officials  is  that  test  results  that  occurred  prior  to  October  2006 
had  yet  to  be  fully  recorded  in  the  contractor’s  tracking  tool. 

DIO  and  contractor  officials  attributed  the  absence  of  adequate 
requirements  traceability  to  the  ongoing  instability  in  requirements  and 
magnitude  of  the  effort  to  update  the  chain  of  preexisting  and  new 
requirements  documentation.  They  added  that  they  expect  traceability  to 
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improve  as  requirements  become  more  stable  and  the  documentation  is 
updated.  Regardless,  the  DIO  has  and  continues  to  invest  in  the 
development  of  DRRS  in  the  absence  of  requirements  traceability.  Without 
traceable  requirements,  the  DIO  does  not  have  a  sufficient  basis  for 
knowing  that  the  scope  of  the  design,  development,  and  testing  efforts  will 
produce  a  system  solution  on  time  and  on  budget  and  that  will  meet  users’ 
operational  needs  and  perform  as  intended.  As  a  result,  the  risk  is 
significant  that  expensive  and  time-consuming  system  rework  will  be 
required. 


Changes  to  Requirements  Adopting  a  disciplined  process  for  reviewing  and  accepting  changes  to  an 
Are  Not  Being  Effectively  approved  and  authoritative  baseline  set  of  requirements  in  light  of  the 
Controlled  estimated  costs,  benefits,  and  risk  of  each  proposed  change  is  a 

recognized  best  practice.  Elements  of  a  disciplined  process  include:  (1) 
formally  documenting  a  requirements  change  process;  (2)  adopting 
objective  criteria  for  considering  proposed  changes,  such  as  estimated 
cost  or  schedule  impact;  and  (3)  rigorously  adhering  to  the  documented 
change  control  process. 

Since  the  inception  of  the  program  in  2002,  DRRS  has  been  developed  and 
managed  without  a  formally  documented  and  approved  process  for 
managing  changes  to  system  requirements.  Further,  while  requirements 
management  and  change  control  plans  were  developed  in  2006  by  the 
DRRS  software  development  contractor,  according  to  DIO  officials,  the 
plans  were  not  adequate.  For  example,  the  plans  did  not  detail  how  DRRS 
user  requirements  were  collected  or  how  objective  factors,  such  as  cost, 
impacted  development  decisions. 

To  address  these  problems,  the  Joint  Staff  developed  what  it  referred  to  as 
a  conceptual  requirements  change  control  process  in  February  2008, 
which  was  to  serve  as  a  basis  for  the  DIO  to  develop  more  detailed  plans 
that  could  be  implemented.  Eleven  months  later,  in  January  2009,  the  DIO 
drafted  more  detailed  plans — a  DRRS  requirements  management  plan  and 
a  DRRS  requirements  configuration  management  plan,  the  latter  of  which 
the  DIO  updated  in  March  2009.  Specifically,  the  draft  plans  call  for  new 
DRRS  requirements  to  be  collected  using  an  online  tool  and  reviewed  by 
the  DIO  to  determine  whether  the  requirement  constitutes  a  major  change 
to  DRRS.  Once  approved,  the  DIO  and  the  contractor  are  to  provide  the 
Battle  Staff  with  a  formatted  report  specifying  the  anticipated  benefit  of 
each  new  requirement  and  an  initial  analysis  of  the  cost  and  performance 
impact.  The  Battle  Staff  then  is  to  prioritize  the  requirement  based  on  the 
DIO’s  impact  analysis.  If  the  issue  cannot  be  resolved  by  the  Battle  Staff,  it 
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is  to  be  elevated  to  the  senior  oversight  bodies  (i.e.,  the  General  Officer’s 
Steering  Committee  and  the  DRRS  Executive  Committee).  After  a 
requirement  has  been  approved,  the  software  developer  may  prepare  a 
more  detailed  “customer  acceptance  document”  that  analyzes  the  potential 
cost,  schedule,  and  quality  impact  to  DRRS  objectives,  which  is  then  to  be 
reviewed  by  the  DIO  at  subsequent  Change  Control  Board  meetings. 

However,  according  to  the  user  community  and  the  DIO  Director,  the 
revised  plans  have  not  been  submitted  for  review  and  approval  to  the 
DRRS  community.  Specifically,  they  stated  that  only  a  proposed  process 
flow  diagram  was  briefed  at  the  Battle  Staff,  and  according  to  them,  the 
change  control  process  was  still  being  evaluated.  Moreover,  the  DIO  has 
yet  to  implement  key  aspects  of  its  draft  plans.  For  example,  the  DRRS 
Chief  Engineer  stated  that  until  recently,  the  DIO  had  continued  to  accept 
changes  to  DRRS  requirements  that  were  submitted  outside  of  the 
designated  online  tool.  In  addition,  the  reports  that  the  Battle  Staff  are  to 
use  in  making  their  requirement  change  determination  do  not  include  the 
anticipated  benefit  and  estimated  cost  or  schedule  impact  of  new 
requirements.  Rather,  these  reports  only  include  the  estimated  number  of 
hours  necessary  to  complete  work  on  a  proposed  requirement.  Moreover, 
contractor  officials  and  users  from  the  Special  Operations  Command  told 
us  that  cost  or  schedule  impacts  have  rarely  been  discussed  at  the  Battle 
Staff  or  Change  Control  Board  meetings.  Our  analysis  of  minutes  from 
change  control  meetings  confirmed  this.  Furthermore,  the  DRRS  Chief 
Engineer  stated  that  the  customer  acceptance  documents  have  only 
recently  been  used. 

Until  the  DIO  effectively  controls  requirements  changes,  it  increases  the 
risk  of  needed  DRRS  capabilities  taking  longer  and  costing  more  to  deliver 
than  necessary. 
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DRRS  Testing  Has  Not 
Been  Adequately 
Performed  and  Key 
Test  Management 
Structures  and 
Controls  Have  Not 
Been  Established 


Effective  system  testing  is  essential  to  successfully  developing  and 
deploying  systems  like  DRRS.  According  to  DOD  and  other  relevant 
guidance,  system  testing  should  be  progressive,  meaning  that  it  should 
consist  of  a  series  of  test  events  that  first  focus  on  the  performance  of 
individual  system  components,  then  on  the  performance  of  integrated 
system  components,  followed  by  system-level  tests  that  focus  on  whether 
the  system  (or  major  system  increments)  are  acceptable,  interoperable 
with  related  systems,  and  operationally  suitable  to  users.  9  For  this  series 
of  related  test  events  to  be  conducted  effectively,  (1)  each  test  event  needs 
to  be  executed  in  accordance  with  well-defined  test  plans,  (2)  the  results 
of  each  test  event  need  to  be  captured  and  used  to  ensure  that  problems 
discovered  are  disclosed  and  corrected,  and  (3)  all  test  events  need  to  be 
governed  by  a  well-defined  test  management  structure. 


Despite  acquiring  and  partially  deploying  a  subset  of  DRRS  increments, 
the  DIO  cannot  demonstrate  that  it  has  adequately  tested  any  of  these 
system  increments,  referred  to  as  system  releases  and  subreleases. 
Specifically,  (1)  the  test  events  for  already  acquired,  as  well  as  currently 
deployed  and  operating,  DRRS  releases  and  subreleases  were  not  based  on 
well-defined  plans,  and  test  events  have  not  been  fully  executed  in 
accordance  with  plans  or  executed  in  a  verifiable  manner,  or  both;  (2)  the 
results  of  executed  test  events  have  not  been  captured  and  used  to  ensure 
that  problems  discovered  were  disclosed  to  decision  makers  and 
ultimately  corrected;  and  (3)  the  DIO  has  not  established  an  effective  test 
management  structure  to  include,  for  example,  a  clear  assignment  of  test 
management  roles  and  responsibilities,  or  a  reliable  schedule  of  planned 
test  events.  Compounding  this  absence  of  test  management  structures  and 
controls  is  the  fact  that  the  DIO  has  yet  to  define  how  the  series  of  system 
releases  and  subreleases  relate  to  its  recent  restructuring  of  DRRS 
increments  into  a  series  of  10  modules.  Collectively,  this  means  that  it  is 
unlikely  that  already  developed  and  deployed  DRRS  capabilities  can 
perform  as  intended  and  meet  user  operational  needs.  Equally  doubtful  are 
the  chances  that  the  DIO  can  adequately  ensure  that  yet-to-be  developed 
DRRS  capabilities  will  meet  expectations. 


9See  for  example,  Department  of  Defense  Instruction  5000.02,  Operation  of  the  Defense 
Acquisition  System-,  Defense  Acquisition  University,  Test  and  Evaluation  Management 
Guide;  Institute  of  Electrical  and  Electronics  Engineers,  Standard  1012-2004 for  Software 
Verification  and  Validation;  and  Software  Engineering  Institute,  Capability  Maturity 
Model  Integration  for  Acquisition  version  1.2  (Pittsburgh,  Pa.:  November  2007). 
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DIO  Has  Not  Adequately 
Planned  and  Executed  Key 
Test  Events 


Key  tests  required  for  already  developed  and  partially  fielded  DRRS 
increments  either  did  not  have  well-defined  test  plans,  or  these  tests  have 
yet  to  be  conducted.  According  to  program  documentation,  system 
releases  and  subreleases  have  been  subjected  to  what  are  described  as  30- 
day  test  cycles,  during  which:  (1)  a  Software  Test  Plan  is  updated  if 
applicable,  (2)  test  procedures  are  developed  and  incorporated  in  the 
Software  Test  Description,  (3)  a  series  of  developmental  tests  on  each 
release/subrelease  is  performed,  (4)  weekly  meetings  are  held  to  review 
software  defects  identified  during  testing,  (5)  final  test  results  are 
summarized  within  the  Software  Test  Report  and  Software  Version 
Description,  and  (6)  the  release/subrelease  is  made  available  to  users. 

However,  the  program  office  has  yet  to  provide  us  with  the  developmental 
test  plans  and  procedures  for  each  release/subrelease  that  to-date  has 
either  been  developed  or  fielded.  Instead,  it  provided  us  with  a  Software 
Test  Plan  and  two  Software  Test  Descriptions  that  it  said  applied  to  two 
subreleases  within  release  4.0.  However,  available  information  indicates 
that  DRRS  subreleases  total  at  least  63,  which  means  that  we  have  yet  to 
receive  the  test  plans  and  procedures  for  61.  Further,  the  test  plan  that  we 
were  provided  is  generic  in  nature,  meaning  that  it  was  not  customized  to 
apply  specifically  to  the  two  subreleases  within  Release  4.0.  Moreover,  the 
plan  and  procedures  lack  important  elements  specified  in  industry 
guidance.10  For  example,  the  test  plan  does  not  include  a  schedule  of 
activities  to  be  performed  or  defined  roles  and  responsibilities  for 
performing  them.  Also,  the  test  plan  does  not  consistently  include  test 
entrance  and  exit  criteria,  a  test  defect  management  process,  and  metrics 
for  measuring  progress. 

Moreover,  the  DIO  has  yet  to  demonstrate  that  it  has  performed  other  key 
developmental  and  operational  test  events  that  are  required  before  the 
software  is  fielded  for  operational  use.  According  to  DIO  officials, 
developmental  testing  concludes  only  after  system  integration  testing  and 
system  acceptance  testing,  respectively,  are  performed.  Further,  following 
developmental  testing,  the  Joint  Interoperability  Test  Command  (JITC), 
which  is  a  DOD  independent  test  organization,  is  to  conduct  both 
interoperability  and  operational  testing  before  the  system  is  deployed  and 
put  into  production  (i.e.,  used  operationally).  Although  increments  of 


10See  for  example,  Institute  of  Electrical  and  Electronics  Engineers,  Standard  829-2008 
Standard  for  Software  and  System  Test  Documentation  (New  York,  N.Y.:  July  18,  2008) 
and  Software  Engineering  Institute,  Capabili  ty  Maturity  Model  Integration  for 
Acquisition,  version  1.2. 
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DRRS  functionality  have  been  put  into  production,  the  DIO  has  not 
performed  system  integration  testing,  system  acceptance  testing,  or 
operational  testing  on  any  DRRS  release  or  subrelease.  Further,  JITC 
documentation  shows  that  while  an  interoperability  test  of  an  increment 
of  DRRS  functionality  known  as  ESORTS  was  conducted,  this  test  did  not 
result  in  an  interoperability  certification. 11  According  to  JITC  and  Joint 
Staff  officials,  this  was  because  the  DIO  did  not  address  JITC’s  identified 
limitations  to  the  program’s  Information  Support  Plan,  which  identifies 
essential  information-exchange  sharing  strategies  between  interdependent 
systems  that  are  needed  for  interoperability  certification.  Without 
interoperability  certification,  the  ability  of  the  DRRS  to  exchange  accurate 
and  timely  readiness  data  with  other  critical  systems,  such  as  the  Joint 
Operation  Planning  and  Execution  System,  cannot  be  ensured. 

Similarly,  while  DIO  officials  stated  that  acceptance  testing  has  occurred 
for  one  increment  of  DRRS  functionality  known  as  SORTSREP,  the  DIO 
does  not  have  either  a  finalized  acceptance  test  plan  or  documented  test 
results.  12  Furthermore,  the  integrated  master  schedule  (last  updated  in 
April  2009)  shows  that  acceptance  testing  is  not  to  occur  until  the 
July/August  2009  time  frame,  which  is  about  15  months  later  than 
originally  envisioned.  Moreover,  this  delay  in  acceptance  testing  has  in 
turn  delayed  interoperability  and  operational  testing  by  16  months 
(May/June  2008  to  September/November  2009),  according  to  the  latest 
schedule.  Program  officials  attributed  the  delays  to  Marine  Corps  and  Air 
Force  concerns  about  the  quality  of  SORTSREP. 

Until  the  DIO  has  effectively  planned  and  executed  the  series  of  tests 
needed  to  demonstrate  the  readiness  of  DRRS  increments  to  operate  in  a 
production  environment,  the  risk  of  fielded  system  increments  not 
performing  as  intended  and  requiring  expensive  rework  to  correct  will  be 
increased,  and  DOD  will  continue  to  experience  delays  in  delivering 
mission-critical  system  capabilities  to  its  readiness  community. 


^According  to  the  Concept  of  Operations,  ESORTS  is  to  provide  current  readiness  status 
for  operational  forces  and  defense  support  organizations  in  terms  of  their  ability  to  perform 
their  mission  essential  tasks. 

12 According  to  the  draft  SORTSREP  Test  and  Evaluation  Master  Plan,  SORTSREP  is  a  tool 
to  capture  and  input  military  departments’  readiness  data  requirements  into  a  readiness 
reporting  database. 
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DIO  Has  Not  Adequately 
Documented  and  Reported 
Test  Results 


Available  results  of  tests  performed  on  already  developed  and  at  least 
partially  deployed  DRRS  releases/subreleases  show  that  the  test  results 
have  not  been  effectively  captured  and  analyzed,  and  have  not  been  fully 
reported.  Moreover,  test  results  for  other  releases/subreleases  do  not 
exist,  thus  minimizing  the  value  of  any  testing  that  has  been  performed. 
According  to  relevant  guidance,  effective  system  testing  includes 
recording  the  results  of  executing  each  test  procedure  and  test  case  as 
well  as  capturing,  analyzing,  correcting,  and  disclosing  to  decision  makers 
problems  found  during  testing  (test  defects).  13  It  also  includes  ensuring 
that  test  entry  and  exit  criteria  are  met  before  beginning  and  ending, 
respectively,  a  given  test  event. 

The  DIO  does  not  have  test  results  of  all  developed  and  tested  DRRS 
releases  and  subreleases.  Specifically,  program  officials  provided  us  with 
the  Software  Test  Reports  and  Software  Version  Descriptions  that,  based 
on  program  documentation,  represent  the  full  set  of  test  results  for  three 
subreleases  and  a  partial  set  of  test  results  for  40  subreleases  within 
releases  1.0,  3.0,  and  4.0.  However,  as  noted  earlier,  DRRS  subreleases 
total  at  least  63,  which  means  that  test  reports  and  results  for  at  least  20 
subreleases  do  not  exist.  Moreover,  the  test  reports  and  version 
descriptions  that  we  received  do  not  consistently  include  key  elements 
provided  for  in  industry  guidance,  such  as  a  documented  assessment  of 
system  capabilities  and  limitations,  entrance/exit  criteria  status,  an 
assessment  as  to  whether  the  applicable  requirements/thresholds  were 
met,  and  unresolved  defects  and  applicable  resolution  plans.  14  This 
information  is  important  because  it  assists  in  determining  and  disclosing 
to  decision  makers  current  system  performance  and  efforts  needed  to 
resolve  known  problems,  and  provides  program  officials  with  a  needed 
basis  for  ensuring  that  a  system  increment  is  ready  to  move  forward  and 
be  used.  Without  this  information,  the  quality  and  readiness  of  a  system  is 
not  clear. 

Furthermore,  the  DIO  does  not  have  detailed  test  defect  documentation 
associated  with  all  executed  DRRS  test  events.  According  to  relevant 


13See  for  example,  Defense  Acquisition  University,  Defense  Acquisition  Guidebook 
(Arlington,  Va.:  November  2006);  Institute  of  Electrical  and  Electronics  Engineers, 
Standard  1012-2004  for  Software  Verification  and  Validation;  Software  Engineering 
Institute,  Capability  Maturity  Model  Integration  for  Acquisition,  version  1.2. 

14See  for  example,  Institute  of  Electrical  and  Electronics  Engineers,  Standard  829-2008 
Standard  for  Software  and  System  Test  Documentation  and  Software  Engineering 
Institute,  Capability  Maturity  Model  Integration  for  Acquisition  version  1.2. 
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guidance,  defect  documentation  should,  among  other  things,  identify  each 
issue  discovered,  assign  each  a  priority/criticality  level,  and  provide  for 
each  a  strategy  for  resolution  or  mitigation.  15  In  lieu  of  detailed  test  defect 
documentation,  program  officials  referred  us  to  the  above-mentioned 
Software  Version  Descriptions,  and  stated  that  additional  information  is 
available  in  an  automated  tool,  known  as  the  ISI  BugTracker,  that  it  uses 
to  capture,  among  other  things,  defect  data.  However,  these  documents  do 
not  include  the  above-cited  defect  information,  and  defect  data  for  each 
test  event  do  not  exist  in  the  ISI  BugTracker. 

Compounding  the  absence  and  limitations  of  test  results  are  weaknesses 
in  the  program  office’s  process  for  collecting  such  results  during  test 
execution.  According  to  relevant  guidance,  test  results  are  to  be  collected 
and  stored  according  to  defined  procedures  and  placed  under  appropriate 
levels  of  control.  16  Furthermore,  these  test  results  are  to  be  reviewed 
against  the  source  data  to  ensure  that  they  are  complete,  accurate,  and 
current.  For  DRRS,  the  program  office  is  following  a  partially 
undocumented,  manual  process  for  collecting  and  storing  test  results  and 
defects  that  involves  a  database  and  layers  of  documentation.  As 
explained  by  program  officials  and  documentation,  the  DIO  initially 
documents  defects  and  completed  test  case  results  manually  on  paper 
forms,  and  once  the  defect  is  approved  by  the  test  lead,  it  is  input  into  a 
database.  However,  it  does  not  have  written  procedures  governing  the 
entire  process,  and  thus  key  controls,  such  as  assigned  levels  of  authority 
for  database  read/write  access,  are  not  clearly  defined.  Moreover,  once  the 
test  results  and  defects  are  input  into  the  database,  traceability  back  to  the 
original  test  data  for  data  integrity  checks  cannot  be  established  because 
the  program  office  does  not  retain  these  original  data  sets.  Program 
officials  acknowledged  these  internal  control  weaknesses  and  stated  that 
they  intend  to  adopt  a  new  test  management  tool  that  will  allow  them  to 
capture  in  a  single  database  test  cases,  test  results,  and  test  defects. 

Furthermore,  the  DIO’s  process  for  analyzing  and  resolving  test  defects 
has  limitations.  According  to  relevant  guidance  and  the  draft  SORTSREP 
Test  and  Evaluation  Master  Plan  (TEMP),  defects  should  be  analyzed  and 


15See  for  example,  Institute  for  Electrical  and  Electronics  Engineers,  Standard  829-2008 
Standard  for  Software  and  System.  Documentation. 

16See  for  example,  Software  Engineering  Institute,  Capability  Maturity  Model  Integration 
for  Acquisition,  version  1.2. 
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prioritized.  17  However,  the  program  office  has  not  established  a  standard 
definition  for  defect  priority  levels  identified  during  testing.  For  example, 
the  various  release/subrelease  test  reports  (dated  through  January  2009) 
prioritize  defects  on  a  scale  of  1-3,  where  a  priority  2  means  critical  but 
with  a  viable  workaround.  In  contrast,  the  SORTSREP  TEMP  (dated 
January  2009)  prioritizes  defects  on  a  scale  of  1-5,  where  a  priority  2 
means  an  error  that  adversely  affects  the  accomplishment  of  an 
operational  or  mission  essential  function  in  accordance  with  official 
requirements  so  as  to  degrade  performance  and  for  which  no  alternative 
work  around  solution  exists.  By  not  using  standard  priority  definitions  for 
categorizing  defects,  the  program  office  cannot  ensure  that  it  has  an 
accurate  and  useful  understanding  of  the  scope  and  magnitude  of  the 
problems  it  is  facing  at  any  given  time,  and  it  will  not  know  if  it  is 
addressing  the  highest  priority  issues  first. 

In  addition,  the  DIO  has  not  ensured  that  critical  defects  are  corrected 
prior  to  concluding  a  given  test  event.  According  to  relevant  guidance  and 
the  draft  SORTSREP  TEMP,  all  critical  and  high  defects  should  be 
resolved  prior  to  the  conclusion  of  a  test  event,  and  all  test  results  should 
be  reviewed  for  validity  and  completeness.18  However,  the  DRRS 
release/subrelease  test  reports  show  that  the  DIO  concluded  five  test 
events  even  though  each  had  at  least  11  open  critical  defects  (priority  1 
defects  with  no  workaround).  Moreover,  these  numbers  of  open  critical 
defects  are  potentially  higher  because  they  do  not  include  defects  for 
which  a  solution  was  identified  but  the  solution  failed  during  regression 
testing  and  do  not  include  defects  that  were  dismissed  because  the 
program  official  was  unable  to  recreate  it. 

Until  the  DIO  adequately  documents  and  reports  the  test  results,  and 
ensures  that  severe  problems  discovered  are  corrected  prior  to  concluding 
a  given  test  event,  the  probability  of  incomplete  test  coverage,  and 


17See  for  example,  GAO,  Office  of  Personnel  Management:  Improvements  Needed  to 
Ensure  Successful  Retirement  Systems  Modernization,  GAO-08-345  (Washington,  D.C.: 
January  2008);  Institute  of  Electrical  and  Electronics  Engineers,  Standard  1012-2004 for 
Software  Verification  and  Validation-,  and  Institute  of  Electrical  and  Electronics 
Engineers,  Standard  1012-1986 for  Softivare  Verification  and  Validation  Plans  (New 
York,  N.Y.:  Nov.  14,  1986). 

18See  for  example,  GAO-08-345;  Institute  of  Electrical  and  Electronics  Engineers,  Standard 
1012-2004 for  Software  Verification  and  Validation;  and  Institute  of  Electrical  and 
Electronics  Engineers,  Standard  1012-1986 for  Software  Verification  and  Validation 
Plans. 
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insufficient  and  invalid  test  results,  is  increased,  thus  unnecessarily 
increasing  the  risk  of  DRRS  not  meeting  mission  needs  or  otherwise  not 
performing  as  intended. 


DIO  Has  Not  Established  The  DIO  does  not  have  an  effective  test  management  structure,  to  include 
an  Effective  Test  a  well-defined  overall  test  management  plan  that  clearly  assigns  test 

Management  Structure  management  roles  and  responsibilities,  a  designated  test  management  lead 

°  and  a  supporting  working  group,  and  a  reliable  schedule  of  planned  test 

events.  According  to  relevant  guidance,  these  aspects  of  test  management 
are  essential  to  adequately  planning,  executing,  and  reporting  a  program’s 
series  of  test  events.  19 

Although  the  program  has  been  underway  for  8  years,  it  did  not  have  an 
overarching  DRRS  TEMP  until  very  recently  (February  2009),  and  this  plan 
is  still  in  draft  and  has  yet  to  be  approved.  Further,  this  draft  TEMP  does 
not  clearly  define  DRRS  test  management  roles  and  responsibilities,  such 
as  those  of  the  test  manager,  and  it  does  not  include  a  reliable  schedule  of 
test  events  that  reflect  the  program’s  recent  restructuring  of  its  software 
releases/subreleases  into  10  modules.  According  to  DIO  officials,  they 
recently  decided  not  to  approve  this  overarching  TEMP.  Instead,  they  said 
that  they  now  intend  to  have  individual  TEMPs  for  each  of  the  recently 
defined  10  modules,  and  to  have  supporting  test  plans  for  each  module’s 
respective  developmental  and  operational  test  events.  According  to 
program  documentation,  three  individual  TEMPs  are  under  development 
(i.e.,  SORTSREP  tool  and  the  Mission  Readiness  and  Readiness  Review 
modules).  However,  drafts  of  these  TEMPs  also  do  not  clearly  define  test 
entrance  and  exit  criteria,  test  funding  requirements,  an  integrated  test 
program  schedule,  and  the  respective  test  management  roles  and 
responsibilities.  For  example,  while  the  draft  SORTSREP  TEMP  identifies 
the  roles  and  responsibilities  of  some  players,  such  as  the  test  manager, 
the  personnel  or  organization  that  is  to  be  responsible  is  not  always 
identified.  In  addition,  while  the  various  players  in  the  user  community  are 
identified  (i.e.,  military  services,  combatant  commands),  their  associated 
roles  or  responsibilities  are  not. 


19See  for  example,  Department  of  Defense  Instruction  5000.02,  Operation  of  the  Defe?ise 
Acquisition  System;  Defense  Acquisition  University,  Defense  Acquisition  Guidebook; 
Software  Engineering  Institute,  Capability  Maturity  Model  Integration  for  Acquisition, 
version  1.2;  Institute  for  Electrical  and  Electronics  Engineers,  Standard  829-2008 
Softivare  and  System .  Test  Documentation  (New  York,  N.Y.:  July  18,  2008). 
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Furthermore,  the  DIO  has  yet  to  designate  a  test  management  lead  and 
establish  an  effective  test  management  working  group.  According  to 
relevant  guidance,  test  management  responsibility  and  authority  should  be 
assigned  to  an  individual,  and  this  individual  should  be  supported  by  a 
working  integrated  product  team  that  includes  program  office  and 
operational  testing  representatives.20  Among  other  things,  the  working 
integrated  product  team  is  to  develop  an  overall  system  test  strategy. 
However,  DIO  officials  told  us  that  the  test  manager  position  has  been 
vacant,  and  this  position  is  now  being  temporarily  filled  by  the  program’s 
chief  engineer,  who  is  a  contractor.  Furthermore,  although  DRRS  system 
development  began  prior  to  2004,  a  charter  for  a  test  and  evaluation 
working  integrated  product  team  was  not  issued  until  February  2009. 
According  to  DIO  officials,  the  delay  in  establishing  the  team  has  not  had 
any  impact  because  of  corresponding  delays  in  finalizing  the  program’s 
overall  test  strategy.  However,  this  statement  is  not  consistent  with  the 
Defense  Acquisition  Guidebook,  which  states  that  two  of  the  key  products 
of  the  working  integrated  product  team  are  the  program’s  test  strategy  and 
TEMP.  Further,  JITC  officials  stated  that  the  lack  of  a  test  manager  and  an 
active  test  and  evaluation  working  integrated  product  team  have  reduced 
the  effectiveness  of  DRRS  testing  activities.  As  a  result,  they  stated  that 
they  have  had  to  compensate  by  conducting  individual  meetings  with  the 
user  community  to  discuss  and  receive  documentation  to  support  their 
operational  and  interoperability  test  planning  efforts. 

Moreover,  the  DIO  has  yet  to  establish  a  reliable  schedule  of  planned  test 
events.  For  example,  the  schedule  in  the  TEMPs  is  not  consistent  with 
either  the  integrated  master  schedule  or  the  developmental  test  plans. 
Specifically,  the  draft  SORTSREP  TEMP  (last  updated  in  January  2009) 
identifies  SORTSREP  developmental  testing  occurring  through  January 
2009  and  ending  in  early  February  2009,  while  the  integrated  master 
schedule  (last  updated  in  April  2009)  shows  SORTSREP  development 
testing  as  occurring  in  the  July/August  2009  time  frame.  In  addition,  while 
program  officials  said  that  development  testing  for  SORTSREP  has 
occurred,  the  associated  development  test  plans  (e.g.,  system  integration 
and  system  acceptance  test  plans)  had  no  established  dates  for  test 
execution,  and  are  still  in  draft.  As  another  example,  a  module  referred  to 
as  “Mission  Readiness”  had  no  established  dates  for  test  execution  in  its 


20See  for  example,  Defense  Acquisition  University,  Defense  Acquisition  Guidebook ; 
Software  Engineering  Institute,  Capability  Maturi  ty  Model  Integration  for  Acquisition, 
version  1.2;  Institute  for  Electrical  and  Electronics  Engineers,  Standard  829-2008 
Softivare  and  System.  Test  Documentation. 
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TEMP,  and  while  program  documentation  indicates  that  this  module 
completed  development  testing  in  December  2008,  the  associated 
development  test  plans  (e.g.,  system  integration  and  system  acceptance 
test  plans)  do  not  exist.21 

In  addition,  the  DIO  has  yet  to  define  in  its  draft  TEMPs  how  the 
development  and  testing  to  date  of  at  least  63  subreleases  relate  to  the 
development  and  testing  of  the  recently  established  10  system  modules. 
According  to  Joint  Staff  and  JITC  officials,  they  do  not  know  how  the 
releases/subreleases  relate  to  the  modules,  and  attributed  this  to  a  lack  of 
an  approved  description  for  each  module  that  includes  what  functionality 
each  is  intended  to  provide.  Furthermore,  the  high-level  schedule  in  the 
TEMP  does  not  describe  what  test  events  for  the  DRRS 
releases/subreleases  that  have  already  been  developed  and  deployed  relate 
to  the  development  test  efforts  planned  for  the  respective  modules.  These 
problems  in  linking  release/subrelease  test  events  to  module  test  events 
limit  the  DIO  and  JITC  in  leveraging  the  testing  already  completed,  which 
in  turn  will  impact  the  program’s  ability  to  meet  cost,  schedule,  and 
performance  expectations. 

Collectively,  the  weaknesses  in  this  program’s  test  management  structure 
increase  the  chances  that  the  deployed  system  will  not  meet  certification 
and  operational  requirements,  and  will  not  perform  as  intended. 


DRRS  Has 
Guided  by 
Schedule 


Not  Been 
a  Reliable 


The  success  of  any  program  depends  in  part  on  having  a  reliable  schedule 
that  defines,  among  other  things,  when  work  activities  will  occur,  how 
long  they  will  take,  and  how  they  are  related  to  one  another.  As  such,  the 
schedule  not  only  provides  a  road  map  for  the  systematic  execution  of  a 
program,  but  also  provides  the  means  by  which  to  gauge  progress,  identify 
and  address  potential  problems,  and  promote  accountability.  From  its 
inception  in  2002  until  November  2008,  the  DIO  did  not  have  an  integrated 
master  schedule.  Moreover,  the  only  milestone  that  we  could  identify  for 
the  program  prior  to  November  2008  was  the  date  that  DRRS  was  to 
achieve  full  operational  capability,  which  was  originally  estimated  to 
occur  in  fiscal  year  2007,  but  later  slipped  to  fiscal  year  2008  and  then 
fiscal  year  2011,  and  is  now  fiscal  year  2014 — a  7-year  delay. 


21According  to  the  Mission  Readiness  TEMP,  Mission  Readiness  is  to  create  a  common 
standard  metric  for  assessing  readiness  by  capability  that  can  be  uniformly  applied 
throughout  the  department. 
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In  addition,  the  DRRS  integrated  master  schedule  that  was  developed  in 
November  2008,  and  was  updated  in  January  2009  and  again  in  April  2009 
to  address  limitations  that  we  identified  and  shared  with  the  program 
office,  is  still  not  reliable.  Specifically,  our  research  has  identified  nine 
practices  associated  with  developing  and  maintaining  a  reliable  schedule.22 
These  practices  are  (1)  capturing  all  key  activities,  (2)  sequencing  all  key 
activities,  (3)  assigning  resources  to  all  key  activities,  (4)  integrating  all 
key  activities  horizontally  and  vertically,  (5)  establishing  the  duration  of  all 
key  activities,  (6)  establishing  the  critical  path  for  all  key  activities,  (7) 
identifying  float  between  key  activities,  (8)  conducting  a  schedule  risk 
analysis,  and  (9)  updating  the  schedule  using  logic  and  durations  to 
determine  the  dates  for  all  key  activities.23  However,  the  program’s  latest 
integrated  master  schedule  does  not  address  three  of  the  practices  and 
only  partially  addresses  the  remaining  six.  For  example,  the  schedule  does 
not  establish  a  critical  path  for  all  key  activities,  include  a  schedule  risk 
analysis,  and  it  is  not  being  updated  using  logic  and  durations  to  determine 
the  dates  for  all  key  activities.  Further,  it  does  not  fully  capture,  sequence, 
and  establish  the  duration  of  all  key  work  activities;  fully  assign  resources 
to  all  key  work  activities;  fully  integrate  all  of  these  activities  horizontally 
and  vertically;  and  fully  identify  the  amount  of  float — the  time  that  a 
predecessor  activity  can  slip  before  the  delay  affects  successor  activities — 
between  these  activities.  These  practices  are  fundamental  to  producing  a 
sufficiently  reliable  schedule  baseline  that  can  be  used  to  measure 
progress  and  forecast  slippages.  (See  table  3  for  the  results  of  our  analyses 
relative  to  each  of  the  nine  practices.) 


22GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 

23Float  is  the  amount  of  time  an  activity  can  slip  before  affecting  the  critical  path. 
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Table  3:  DRRS  Satisfaction  of  Nine  Schedule-Estimating  Key  Practices 


Practice 


Explanation 


Practice  met?  GAO  analysis 


Capturing  all  activities  The  schedule  should  reflect  all  key  activities  Partially 

(e.g.,  steps,  events,  outcomes,  etc.)  as 
defined  in  the  program's  work  breakdown 
structure,  to  include  activities  to  be  performed 
by  both  the  government  and  its  contractors 


The  schedule  reflects  many  key  activities 
as  defined  in  the  program's  work 
breakdown  structure.  However,  key 
activities  are  identified  only  as  milestones 
rather  than  in  terms  of  work  to  be 
performed  and  accomplished  to  achieve 
the  milestones.  Thus,  the  schedule  does 
not  account  for  all  work  to  be  performed. 
As  a  result,  the  reliability  of  the  milestones 
is  questionable. 


Sequencing  all  activities  The  schedule's  activities  need  to  be  logically  Partially 

sequenced  in  the  order  that  they  are  to  be 
carried  out.  In  particular,  activities  that  must 
finish  prior  to  the  start  of  other  activities  (i.e., 
predecessor  activities)  as  well  as  activities 
that  cannot  begin  until  other  activities  are 
completed  (i.e.,  successor  activities)  should 
be  identified.  By  doing  so,  interdependencies 
among  activities  that  collectively  lead  to  the 
accomplishment  of  key  events  or  milestones 
can  be  established  and  used  as  a  basis  for 
guiding  work  and  measuring  progress. 


The  schedule  identifies  some  predecessor 
and  successor  activities,  but  not  all. 
Specifically,  out  of  290  key  activities,  139 
do  not  identify  predecessor  activities  and 
121  do  not  identify  successor  activities. 
DIO  officials  stated  that  this  is  because 
interdependencies  are  discussed  and 
addressed  at  various  meetings,  and  thus 
do  not  need  to  be  in  the  schedule. 
However,  recognition  of  such 
interdependencies  in  the  schedule  is 
necessary  to  logically  link  tasks  and 
events  and  thereby  calculate  dates  and 
predict  changes  in  the  future.  Without 
proper  linkages,  tasks  that  slip  early  in  the 
schedule  do  not  transmit  delays  to  tasks 
that  should  depend  on  them.  By  not 
logically  linking  all  key  activities,  the 
schedule  does  not  provide  a  sufficient 
basis  for  understanding  the  program  as  a 
whole,  and  confidence  that  the  right  dates 
or  critical  paths  are  represented  is  low. 
This  means  thatthe  DIO  cannot  use  its 
schedule  to  identify  disconnects  as  well  as 
hidden  opportunities,  and  cannot 
otherwise  promote  efficiency  and 
accuracy,  and  control  the  program  by 
comparing  actual  to  planned  progress. 
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Practice 

Assigning  resources  to  all 
activities 


Establishing  the  duration 
of  all  activities 


Explanation  Practice  met?  GAO  analysis 

The  schedule  should  realistically  reflect  what  Partially  The  schedule  identifies  15  resources  that 

resources  (i.e.,  labor,  material,  and  are  assigned  to  various  activities, 

overhead)  are  needed  to  do  the  work,  However,  important  information  is  not 

whether  all  required  resources  will  be  included,  which  hampers  its  usefulness, 

available  when  they  are  needed,  and  whether  For  example,  one  benefit  of  loading 

any  funding  or  time  constraints  exist.  resource  information  is  to  ensure  that 

resources  in  short  supply  will  not  be 
overscheduled  in  any  time  period. 
However,  the  schedule  does  not  define  the 
amount  of  each  resource  that  is  available, 
but  instead  assumes  only  one  unit  of  each 
resource  is  available.  As  a  result,  10  of  the 
15  types  of  resources  (e.g.,  information 
assurance  and  test  and  evaluation)  in  the 
schedule  are  overscheduled  and  thus 
estimated  completion  dates  are  not 
realistic.  Further,  the  schedule  does  not 
identify  the  costs  of  the  resources. 

Knowing  the  cost  of  resources  is 
important,  because  some  resources,  such 
as  labor,  can  cost  more  if  the  program 
takes  longer. 


The  schedule  should  realistically  reflect  how  Partially 

long  each  activity  will  take  to  execute.  In 

determining  the  duration  of  each  activity,  the 

same  rationale,  historical  data,  and 

assumptions  used  for  cost  estimating  should 

be  used  for  schedule  estimating.  Further, 

these  durations  should  be  as  short  as 

possible,  and  they  should  have  specific  start 

and  end  dates.  Excessively  long  periods 

needed  to  execute  an  activity  (i.e.,  greater 

than  2-3  months)  should  prompt  further 

decomposition  of  the  activity  so  that  shorter 

execution  durations  will  result. 


The  schedule  establishes  the  duration  of 
key  activities  and  includes  specific  start 
and  end  dates.  However,  the  activities  are 
not  always  of  short  duration.  For  example, 
23  activities  have  remaining  durations  that 
exceed  100  days  (108  to  551  days).  By 
having  a  schedule  summarized  attoo  high 
a  level,  the  program  runs  the  risk  of 
masking  critical  work  elements  within  the 
key  activities  associated  with  executing  the 
program  and  fails  to  show  the  risk 
management  approaches  being  used. 
Further,  it  risks  masking  the  schedule's 
true  critical  path. 


Integrating  schedule  The  schedule  should  be  horizontally  Partially 

activities  horizontally  and  integrated,  meaning  that  it  should  link  the 
vertically  products  and  outcomes  associated  with 

already  sequenced  activities.  These  links  are 
commonly  referred  to  as  "hand  offs"  and 
serve  to  verify  that  activities  are  arranged  in 
the  right  order  to  achieve  aggregated 
products  or  outcomes.  The  schedule  should 
also  be  vertically  integrated,  meaning  that 
traceability  exists  among  varying  levels  of 
activities  and  supporting  tasks  and  subtasks. 

Such  mapping  or  alignment  among  levels 
enables  different  groups  to  work  to  the  same 
master  schedule. 


The  schedule  is  not  horizontally  integrated, 
meaning  that  the  activities  are  not 
arranged  in  the  right  order  to  achieve 
aggregated  products  or  outcomes.  This  is 
because,  as  previously  noted,  many 
activities  do  not  identify  interdependencies 
(predecessor  and  successor  activities).  As 
a  result,  management  lacks  information  on 
how  a  slippage  in  completing  one  activity 
will  affect  others.  In  contrast,  the  schedule 
is  vertically  integrated,  meaning  that  for 
those  key  activities  that  are  identified, 
traceability  exists  among  varying  levels  of 
activities  and  that  lower-level  activities  are 
within  the  constraints  of  higher-level 
activities. 
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Practice  Explanation 

Establishing  the  critical  The  critical  path— the  longest  duration  path 
path  for  all  activities  through  the  sequenced  list  of  key  activities — 

should  be  identified  using  scheduling 
software.  The  establishment  of  a  program's 
critical  path  is  necessary  for  examining  the 
effects  of  any  activity  slipping  along  this  path. 
H  igh-risk  activities  along  or  near  the  critical 
path  should  also  be  identified  and  associated 
time  impacts  of  these  risks  should  be 
reflected  in  the  schedule. 

Identifying  float  between  The  schedule  should  identify  float  time  so 
activities  that  schedule  flexibility  can  be  determined. 

As  a  general  rule,  activities  along  the  critical 
path  typically  have  the  least  amount  of  float 
time. 
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No  The  schedule  does  not  identify  a  valid 

critical  path  because  there  is  no  logical  link 
between  the  program's  key  activities. 
Instead,  the  activities  are  presented  as 
being  independent  from  one  another. 
Without  a  critical  path,  management 
cannot  know  the  activities  critical  to  the  on- 
time  completion  of  DRRS,  orthe  impact  of 
any  changes  to  activities  on  the  path. 

Partially  The  schedule  identifies  the  amount  of  float 

allocated  to  key  activities.  However,  the 
amount  of  float  associated  with  56  of  these 
activities  is  unusually  large  (100  -  461 
days).  Such  large  amounts  of  float  time 
can  be  attributed  to  the  fact  that  many  of 
the  activities  do  not  identify  linkages  to 
other  activities  (predecessor  or  successor 
activities).  In  addition,  the  amount  of  float 
between  activities  is  of  questionable 
accuracy  because  activity  dependencies 
(predecessor  or  successor  activities)  are 
often  not  identified.  Instead,  the  start  and 
completion  dates  for  many  activities  are 
imposed,  rather  than  based  on  logic,  such 
as  an  activity  not  starting  until  its 
predecessor  is  completed.  F urther,  it  is 
unclear  whether  activities  along  the  critical 
path  have  the  least  amount  of  float 
because,  as  previously  noted,  the 
schedule  does  not  have  a  valid  critical 
path. 
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Practice 


Explanation 


Practice  met?  GAO  analysis 


Conducting  a  schedule  A  schedule  risk  analysis  uses  a  good  critical  No 
risk  analysis  path  method  schedule  and  data  about  project 

schedule  risks  as  well  as  Monte  Carlo 
simulation3  techniques  to  predict  the  level  of 
confidence  in  meeting  a  program's 
completion  date,  the  amount  of  time 
contingency  needed  for  a  level  of  confidence, 
and  the  identification  of  high-priority  risks. 

This  analysis  focuses  not  only  on  critical  path 
activities  but  also  on  other  schedule  paths 
that  may  become  critical.  Schedule  reserve 
for  contingencies  should  be  calculated  by 
performing  a  schedule  risk  analysis.  As  a 
general  rule,  the  reserve  should  be  held  by 
the  project  or  program  manager  and  applied 
as  needed  to  those  activities  that  take  longer 
than  scheduled  because  of  the  identified 
risks.  Reserves  of  time  should  not  be 
apportioned  in  advance  to  any  specific 
activity  since  the  risks  that  will  actually  occur 
and  the  magnitude  of  their  impact  is  not 
known  in  advance. 


The  program  office  did  not  perform  a 
schedule  risk  analysis.  Without  this 
analysis,  the  office  cannot  sufficiently 
understand  the  level  of  confidence  in 
meeting  the  program's  completion  date 
and  identifying  reserves  for  contingencies. 


Updating  the  schedule  The  schedule  should  use  logic  and  durations  No 
using  logic  and  durations  in  order  to  reflect  realistic  start  and 
to  determine  the  dates  completion  dates  for  program  activities.  The 

schedule  should  be  continually  monitored  to 
determine  when  forecasted  completion  dates 
differ  from  the  planned  dates,  which  can  be 
used  to  determine  whether  schedule 
variances  will  affect  downstream  work. 

Maintaining  the  integrity  of  the  schedule  logic 
is  not  only  necessary  to  reflect  true  status, 
but  is  also  required  before  conducting  a 
schedule  risk  analysis.  The  schedule  should 
avoid  logic  overrides  and  artificial  constraint 
dates  that  are  chosen  to  create  a  certain 
result  on  paper.  To  ensure  that  the  schedule 
is  properly  updated,  individuals  trained  in 
critical  path  method  scheduling  should  be 
responsible  for  updating  the  schedule's 
status. 


The  realism  of  start  and  completion  dates 
for  some  activities  is  questionable 
because,  as  previously  noted,  some 
activities  do  not  identify  logical 
relationships  (i.e.,  interdependencies  with 
other  activities)  or  are  of  unusually  lengthy 
duration.  In  addition,  there  is  no  "status 
date"  information  in  the  schedule  (i.e., 
evidence  the  overall  schedule  is  updated 
on  a  regular  basis  to  capture  actual  start 
and  completion  dates).  Withoutthis 
information,  the  impact  of  deviations  on 
downstream  work  cannot  be  fully 
understood  and  proactively  addressed.  In 
addition,  some  dates  in  the  schedule  were 
updated  erroneously.  For  example,  the 
schedule  shows  25  activities  as  having 
actual  start  dates  in  the  future,  including  6 
starting  in  2010,  even  though  the  updated 
schedule  was  developed  in  April  2009. 
Likewise,  there  are  12  activities  with  actual 
100  percent  complete  finish  dates  that 
range  from  May  2009  to  J  uly  2010. 


Source:  GAO  analysis  of  DOD  data. 

SA  Monte  Carlo  simulation  allows  the  model’s  parameters  to  vary  simultaneously  according  to  their 
associated  probability  distribution.  The  result  is  a  set  of  estimated  probabilities  of  achieving 
alternative  outcomes,  given  the  uncertainty  in  the  underlying  parameters. 
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The  limitations  in  the  program’s  latest  integrated  master  schedule,  coupled 
with  the  program’s  7-year  slippage  to  date,  make  it  likely  that  DRRS  will 
incur  further  delays.  Compounding  these  limitations  is  the  considerable 
concurrency  in  the  key  activities  and  events  in  the  schedule  associated 
with  the  10  recently  identified  system  modules  (see  fig.  2).  For  example,  in 
2010  alone,  the  program  office  plans  to  complete  development  testing  on  2 
modules  and  operational  testing  on  3  modules,  while  also  reaching  initial 
operational  capability  on  3  modules  and  full  operational  capability  on  2 
modules.  By  way  of  comparison,  the  program  office  had  almost  no 
concurrency  across  a  considerably  more  modest  set  of  activities  and 
events  over  the  last  5  years,  but  nevertheless  has  fallen  7  years  behind 
schedule.  As  previously  reported,  such  significant  overlap  and 
concurrency  among  major  program  activities  can  create  contention  for 
limited  resources  and  thus  introduce  considerable  cost,  schedule,  and 
performance  risks.24 


24GAO,  Information  Technology:  Improvements  for  Acquisition  of  Customs  Trade 
Processing  System  Continue,  but  Further  Efforts  Needed  to  Avoid  More  Cost  and 
Schedule  Shortfalls,  GAO-08-46  (Washington,  D.C.:  Oct.  25,  2007)  and  Secure  Border 
Initiative:  SBInet  Planning  and  Management  Improvements  Needed  to  Control  Risks, 
GAO-07-504T  (Washington,  D.C.:  Feb.  27,  2007). 
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Figure  2:  Schedule  for  Developing  and  Implementing  DRRS  Capabilities 


Modules 

April 

,  2009 

J  oint  Mission 
Readiness 

•  ♦ 

1 

o 

o 

Unit  Mission 
Readiness 

• 

"  4 

5  o 

► 

Asset 

Visibility 

• 

■  «. 

o 

Business 

Intelligence 

• 

♦ 

■  § 

Installation 

Readiness 

• 

♦ 

■ 

o  o 

Readiness 

Reviews 

• 

♦ 

■ 

o 

o 

Planning/ 

Execution  Support 

• 

■ 

♦  ° 

Historical 

Data 

4 

» 

■ 

♦  ° 

System 

Architecture 

• 

o 

End-User 

Usability 

• 

o 

J  |f|m|a|m|j  |j  |a|s|o|n|d 

2004 

1  |f|m|a|m|j  |j  |a|s|o|n|d 

2005 

|f|m|a|m|i  |)  |a|s|o|»|d|]  |f|m|a|m|j  |]  |a|s|o|n|d|j  |f|m|a|m|]  |i  |a|s|o|h|d|i  |f|m|a|m|i  |]  |a|s|o|fi|d 

2006  |  2007  |  2008  1  2009 

#  Begin  planning,  design,  and  development 

■  Complete  developmental  testing  and  evaluation 

O  Complete  operational  testing  and  evaluation 

♦  Complete  initial  operational  capability 

O  Complete  final  operational  capability 

]  |f|m|a|m|j  |j  |a|s|o|n|d 
2010 

|f|m|a|m|j  |j  |a|s|o|n|d 
2011 

|f|m|a|m|j  |j  |a|s|o|n|d 
2012 

J  |f|m|a|m|j  |j  |a|s|o|n|d| 

2013  1 

Source:  GAO  analysis  based  on  DOD  data. 


In  addition,  the  schedule  remains  unstable  as  evidenced  by  the  degree  of 
change  it  has  experienced  in  just  the  past  few  months.  For  example,  the 
January  2009  schedule  had  a  full  operational  capability  milestone  of 
October  2011.  By  contrast,  the  April  2009  schedule  has  a  December  2013 
milestone  (see  fig.  3  below).  Moreover,  some  milestones  are  now  to  occur 
much  earlier  than  they  were  a  few  months  ago.  For  example,  the  January 
2009  schedule  shows  initial  operational  capability  for  “readiness  reviews” 
to  be  June  2010.  However,  the  April  2009  schedule  shows  that  that  this 
milestone  was  attained  in  August  2007.  Overall,  multiple  milestones  for 
four  modules  were  extended  by  at  least  1  year,  including  two  milestones 
that  were  extended  by  more  than  2  years.  Such  change  in  the  schedule  in 
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but  a  few  months  suggests  a  large  degree  of  uncertainty,  and  illustrates  the 
importance  of  ensuring  that  the  schedule  is  developed  in  accordance  with 
best  practices. 


Figure  3:  Changes  in  Estimated  Dates  for  DRRS  Capabilities 
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Source:  GAO  analysis  based  on  DOD  data. 
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DIO  Lacks  Adequate 
Staffing  and  an 
Effective  Approach  to 
Meeting  its  Human 
Capital  Needs 


As  we  have  previously  reported,  effective  human  capital  management  is  an 
essential  ingredient  to  achieving  successful  program  outcomes.25  Among 
other  things,  effective  human  capital  management  involves  a  number  of 
actions  to  proactively  understand  and  address  any  shortfalls  in  meeting  a 
program’s  current  and  future  workforce  needs.  These  include  an 
assessment  of  the  core  competencies  and  essential  knowledge,  skills,  and 
abilities  needed  to  perform  key  program  management  functions,  an 
inventory  of  the  program’s  existing  workforce  capabilities,  and  an  analysis 
of  the  gap  between  the  assessed  needs  and  the  existing  capabilities. 
Moreover,  they  include  explicitly  defined  strategies  and  actions  for  filling 
identified  gaps,  such  as  strategies  for  hiring  new  staff,  training  existing 
staff,  and  contracting  for  support  services. 


The  DIO  is  responsible  for  performing  a  number  of  fundamental  DRRS 
program  management  functions.  For  example,  it  is  responsible  for 
acquisition  planning,  performance  management,  requirements 
development  and  management,  test  management,  contractor  tracking  and 
oversight,  quality  management,  and  configuration  management.  To 
effectively  perform  such  functions,  program  offices,  such  as  the  DIO,  need 
to  have  not  only  well-defined  policies  and  procedures  and  support  tools 
for  each  of  these  functions,  but  also  sufficient  human  capital  to  implement 
the  processes  and  use  the  tools  throughout  the  program’s  life  cycle. 
Without  sufficient  human  capital,  it  is  unlikely  that  a  program  office  can 
effectively  perform  its  basic  program  management  functions,  which  in  turn 
increases  the  risk  that  the  program  will  not  deliver  promised  system 
capabilities  and  benefits  on  time  and  on  budget. 


The  DIO  does  not  currently  have  adequate  staff  to  fulfill  its  system 
acquisition  and  deployment  responsibilities.  In  particular,  the  DIO  is 
staffed  with  a  single  full-time  government  employee — the  DIO  Director.  All 
other  key  program  office  functions  are  staffed  by  either  contractor  staff  or 
staff  temporarily  detailed,  on  an  as-needed  basis,  from  other  DOD 
organizations  (referred  to  as  “matrixed”  staff).  As  a  result,  program 
management  positions  that  the  DIO  itself  has  identified  as  critical  to  the 
program’s  success,  such  as  configuration  manager  and  security  manager, 
are  being  staffed  by  contractors.  Moreover,  these  contractor  staff  report  to 
program  management  positions  also  staffed  by  contractors.  Other  key 
positions,  such  as  those  for  performing  acquisition  management, 


25GAO,  A  Model  of  Strategic  Human  Capital  Management,  GAO-02-373SP  (Washington, 
D.C.:  Mar.  15,  2002). 
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requirements  development  and  management,  and  performance 
management,  have  not  even  been  established  within  the  DIO. 

Furthermore,  key  positions,  such  as  test  manager,  are  vacant.  These 
human  capital  limitations  were  acknowledged  by  the  DRRS  Executive 
Committee  in  November  2008. 

According  to  DIO  and  contractor  officials,  they  recognize  that  additional 
program  management  staffing  is  needed.  They  also  stated  that  while  DRRS 
has  been  endorsed  by  USD  (P&R)  leadership  and  received  funding 
support,  past  requests  for  additional  staff  have  not  been  approved  by  USD 
(P&R)  due  to  other  competing  demands  for  staffing.  Further,  DIO  officials 
stated  that  the  requests  for  staff  were  not  based  on  a  strategic  gap  analysis 
of  its  workforce  needs  and  existing  capabilities.  Specifically,  the  program 
has  not  assessed  its  human  capital  needs  and  the  gap  between  these  needs 
and  its  onboard  workforce  capabilities.  Until  the  program  office  adopts  a 
strategic,  proactive  approach  to  managing  its  human  capital  needs,  it  is 
unlikely  that  it  will  have  an  adequate  basis  for  obtaining  the  people  it 
needs  to  effectively  and  efficiently  manage  DRRS. 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 
4000  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-4000 

SEP  3  2009 

MIWOHHei.  AND 
REAONSS3 

Ms.  Sharon  Pickup 

Director  Defense  Capabilities  and  Management 
U.  S.  Government  Accountability  Office 
Washington,  D.  C.  22548 

Deal-  Ms.  Pickup 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report, 
GAO-09-518,  ‘MILITARY  READINESS:  DoD  Needs  to  Strengthen  Management  and 
Oversight  of  the  Defense  Readiness  Reporting  System,’  dated  July  1,  2009  (GAO  Code 
351217). 

Thank  you  for  the  opportunity  to  review  this  draft  report.  However,  in  our  view, 
the  report  is  flawed  in  its  assessment  of  the  Defense  Readiness  Reporting  System 
(DRRS).  DRRS  is  a  true  net-centric  application  that  provides  broad  and  detailed 
visibility  on  readiness  issues.  Achieving  this  data  sharing  across  the  DoD  enterprise  was 
groundbreaking  work,  fraught  with  barriers  and  obstacles  -  many  of  which  have  now 
been  overcome. 

We  were  disappointed  to  learn  that  the  report  did  not  address  the  cultural 
impediments  that  are  the  root  cause  of  many  of  the  issues  it  cites,  and  the  cause  of  so 
many  previous  congressional  concerns  on  readiness  reporting.  Instead,  the  report  focused 
on  past  issues  and  problems  in  the  software  development  and  acquisition  processes  that 
have  now  been  remedied.  We  believe  that  this  focus,  coupled  with  inaccurate  and 
misleading  factual  information  included  in  the  report,  has  lead  the  GAO  to  develop  an 
incomplete  picture  of  the  program. 

Attached,  please  find  the  Department's  detailed  comments  in  response  to  the 
GAO’s  specific  recommendation. 


Attachment: 
As  stated 


Sincerely, 


William  J.  Can- 

Deputy  Under  Secretary  of  Defense 
(Military  Personnel  Policy) 
Performing  the  Duties  of  the 
Under  Secretary  of  Defense 
(Personnel  and  Readiness) 

© 
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GAO  DRAFT  REPORT  -  DATED  JULY  1, 2009 
GAO  CODE  351217/GAO-09-518 

“MILITARY  READINESS:  DoD  Needs  to  Strengthen  Management  and 
Oversight  of  the  Defense  Readiness  Reporting  System” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  TIIE  RECOMMENDATIONS 

RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense,  as  the  Chair  of  the  Defense  Business  Systems 
Management  Committee,  to  reconsider  the  committee 's  recent  approval  of  the  Defense 
Readiness  Reporting  System  (DRRS)  planned  investment  for  fiscal  years  2009  and  2010, 
and  convene  the  Defense  Business  Systems  Management  Committee  to  review  the 
program's  past  performance  and  the  DRRS  Implementation  Office's  capability  to  manage 
and  deliver  DRRS  going  forward. 

DOD  RESPONSE:  Non-Concur.  The  IRB  and  DBSMC  certifications  were  granted  in 
compliance  with  the  established  certification  process,  which  is  designed  to  not  be  overly 
redundant  with  the  acquisition  process.  Consistent  with  the  Department’s  concept  of 
tiered  accountability,  oversight  of  the  specific  issues  identified  by  the  GAO  in  this  report 
are  the  responsibility  of  component  responsible  for  the  system.  In  this  case,  USD  (P&R) 
chartered  a  DRRS  Executive  Committee  (DEXCOM)  to  provide  for  governance  of  the 
effort  in  the  readiness  community.  The  DEXCOM  has  and  will  continue  to  provide 
appropriate  governance  for  this  effort.  Additionally,  the  USD  (P&R)  will  ensure  that  the 
program  is  compliant  with  all  acquisition  requirements  prior  to  submission  for  further 
certifications. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense  to  have  the  Director  of  the  Business  Transformation 
Agency,  using  the  appropriate  team  of  functional  and  technical  experts  and  the 
established  risk  assessment  methodology,  conduct  a  program  risk  assessment  of  DRRS, 
and  to  use  the  findings  in  GAO'S  report  and  the  risk  assessment  to  decide  how  to  redirect 
the  program's  structure,  approach,  funding,  management,  and  oversight.  In  this  regard, 
GAO  recommends  that  the  Secretary  of  Defense  direct  the  Deputy  Secretary  of  Defense 
to  solicit  the  advice  and  recommendations  of  the  DRRS  Executive  Committee. 

DOD  RESPONSE:  Concur.  The  Department  accepts  the  GAO’s  recommendation  that 
a  program  risk  assessment  would  be  constructive.  The  Business  Transformation  Agency 
(BTA)  will  work  with  the  DRRS  Executive  Committee  (DEXCOM)  to  determine  the 
scope  of  the  assessment  and  will  subsequently  work  with  the  DEXCOM  to  analyze  its 
results.  This  assessment  will  be  complete  by  the  middle  of  Fiscal  Year  20 1 0. 
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RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense  through 
the  appropriate  chain  of  command,  take  steps  to  ensure  that  DRRS  requirements  are 
effectively  developed  and  managed  with  appropriate  input  from  the  Services,  Joint  Staff, 
and  Combatant  Commanders,  including: 

(a)  establishing  an  authoritative  set  of  baseline  requirements  prior  to  further 
system  design  and  development: 

(h)  ensuring  that  the  different  levels  of  requirements  and  their  associated  design 
specifications  and  test  cases,  are  aligned  with  one  another,  and 
(c)  developing  and  instituting  a  disciplined  process  for  reviewing  and  accepting 
changes  to  the  baseline  requirements  in  light  of  estimated  costs,  benefits,  and  risk. 

POD  RESPONSE:  Non-Concur.  The  DRRS  program  has  an  authoritative  set  of 
baseline  requirements  already  established.  The  current  DRRS  governance  process,  which 
includes  membership  from  ali'CoComs,  Services,  Combat  Support  Agencies  (CSAs), 
Joint  Staff,  and  OSD  oversees  the  DRRS  requirements  management  process.  These 
requirements  are  reviewed  bi-weekly  as  part  of  the  DRRS  configuration  control  process. 

RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense  through 
the  appropriate  chain  of  command,  take  steps  to  ensure  that  DRRS  testing  is  effectively 
managed,  including 

(a)  developing  test  plans  and  procedures  for  each  system  increment  test  event  that 
include  a  schedule  of  planned  test  activities,  defined  roles  and  responsibilities, 
test  entrance  and  exit  criteria,  test  defect  management  processes,  and  metrics  for 
measuring  test  progress: 

(b)  ensuring  that  all  key  test  events  are  conducted  on  all  DRRS  increments; 

(c)  capturing,  analyzing,  reporting,  and  resolving  all  test  results  and  test  defects 
of  all  developed  and  tested  DRRS  increments,  and 

(d)  establishing  an  effective  test  management  structure  that  includes  assigned  test 
management  roles  and  responsibilities .  a  designated  test  management  lead  and  a 
supporting  working  group,  and  a  reliable  schedule  of  test  events. 

DoD  RESPONSE:  Non-Concur.  DRRS  testing  is  already  in  place  and  performing 
effectively.  The  DIO  goes  through  a  rigorous  testing  regimen  that  includes  documenting 
test  plans  with  user  test  cases  for  each  incremental  release.  The  DRRS  program  utilizes 
System  Integration  Testing  (SIT),  System  Acceptance  Testing  (SAT),  Interoperability 
Testing  (IOT)  and  Operational  Testing  (OT)  with  System  Integration  Test  Plans  and 
System  Acceptance  Test  Plans.  There  are  designated  testers  independent  of  the 
developers  that  validate  the  user  lest  cases  and  functionality  prior  to  a  deployment 
decision.  Finally,  the  DIO  conducts  a  release  review  with  the  developers  and  the  testers  to 
determine  if  the  increment  is  ready  for  rel  ease  and  meets  all  the  exit  criteria.  For 
interoperability  testing  with  systems  external  to  DRRS.  the  DIO  has  a  designated  test 
director  and  the  joint  Interoperability  Test  Command  (JITC)  is  the  designated 
Interoperability  Test  Activity  and  the  Operational  Test  Activity.  JITC  is  responsible  for 
developing  the  Interoperability  Test  Plan  and  the  Operational  Test  Plan. 
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RECOMMENDATION  5:  The  GAO  recommends  tnat  the  Secretary  of  Defense  through 
the  appropriate  chain  of  command,  take  steps  to  ensure  that  the  DRRS  integrated  master 
schedule  is  reliable,  including  ensuring  that  the  schedule: 

(a)  captures  all  activities  from  the  work  breakdown  structure,  including  the  work 
to  be  performed  and  the  resources  to  be  used, 

(b)  identifies  the  logical  sequencing  of  all  activities,  including  defining 
predecessor  and  successor  activities ; 

(c)  reflects  whether  all  required  resources  will  be  available  when  needed  and 
their  cost; 

(d)  ensures  that  all  activities  and  their  duratitw  are  not  summarized  at  a  level  that 
could  mask  critical  elements; 

(e)  achieves  horizontal  integration  in  the  schedule  by  ensuring  that  all  external 
interfaces  (hand-offs)  are  established  and  ink  rdependencies 

among  activities  are  defined; 

(f)  indentifles  float  between  activities  by  ensuring  that  the  linkages  among  all 
activities  are  defined; 

(g)  defines  a  critical  path  that  runs  continuously  to  the  program's  finish  date; 

(h)  incorporates  the  results  of  a  schedule  risk  analysis  to  determine  the  level  of 
confidence  in  meeting  the  program's  activities  and  completion 

date;  and 

(i)  includes  the  actual  start  and  completion  dates  of  work  activities  performed  so 
that  the  impact  of  deviations  on  downstream  i  vork  can  be  proactively  addressed. 

POD  RESPONSE:  Non-Concur.  A  process  is  already  in  place  to  ensure  the  DRRS 
integrated  master  schedule  is  current,  reliable,  and  meets  all  of  the  criteria  outlined  in  the 
recommendation.  With  the  approval  of  the  DEXCOM  charter  and  the  renewed/refined 
Battle  Staff  and  General  Officer  Steering  Committee  (GOSC),  many  of  the  program 
issues  regarding  Plan  of  Action  and  Milestones  (POA&M)  software  development, 
integration,  and  testing  in  the  GAO  report  are  being  address  or  have  been  corrected. 

RECOMMENDATION  6:  The  GAO  recommends  tnat  the  Secretary  of  Defense  through 
the  appropriate  chain  of  command,  take  steps  to  ensi  re  that  the  DRRS  program  office  is 
staffed  on  the  basis  of  a  human  capital  strategy  that  is  grounded  in  cm  assessment  of  the 
core  competencies  and  essential  knowledge,  skills,  and  abilities  needed  to  perform  key 
DRRS  program  management  functions,  an  inventory  of  the  program  office's  existing 
workforce  capabilities,  and  an  analysis  of  the  gap  be h ween  the  assessed  needs  and  the 
existing  capabilities. 

POD  RESPONSE:  Partially  concur.  The  DIO  was  intentionally  kept  small  in  order  to 
maximize  flexibility  and  responsiveness  to  customer  requirements  as  well  as  to 
implement  commercial  best  practices  for  agile  software  development.  However,  due  to 
increase  demands  on  DRRS  through  additional  requi:  ements  and  capabilities  requested 
by  users,  additional  billets,  both  military  and  civilian  are  needed  to  expedite  the 
implementation  of  DRRS.  To  that  end,  the  USD  (P<&  R)  is  planning  on  adding  full  time 
civilian  acquisition  support  for  the  DIO  and  the  conversion  of  contractor  to  civilian  billets 
during  2010/201 1  timeframe  as  part  of  the  Department's  in-sourcing  initiative. 
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RECOMMENDATION  7:  The  GAO  recommends  tnat  the  Secretary  of  Defense  through 
the  appropriate  chain  of  command,  take  steps  to  ensure  that  DRRS  is  developed  and 
implemented  in  a  manner  that  does  not  increase  the  reporting  burden  on  units  and 
addresses  the  timeliness,  precision,  and  objectivity  oj  metrics  that  are  reported  to 
Congress. 

POD  RESPONSE:  Non-Concur.  One  of  the  primary  tenets  for  DRRS  development  has 
always  been  to  reduce  the  reporting  requirements  on  the  war  fighter.  DRRS  is  already- 
using  state  of  the  art  technology  in  order  to  ensure  that  data  which  already  exists  is 
available  within  the  DRRS  enterprise  in  near-real  litr  e  for  the  war  fighters.  The  DRRS 
governance  structure  that  is  currently  in  place  ensures  that  DRRS  development  docs  not 
deviate  from  these  core  principles. 

RECOMMENDATION  8:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Deputy  Secretary  of  Defense  to  ensure  that  both  die  Human  Resources  Management 
Investment  Review  Board  and  the  DRRS  Executive  Committee  conduct  frequent  oversight 
activities  of.  the  DRRS  program,  and  report  any  significant  issues  to  the  Deputy  Secretary 
of  Defense. 

POD  RESPONSE:  Concur.  The  Department  is  cc  mmitted  to  ensuring  that  the  proper 
level  of  oversight  of  the  DRRS  program  is  adhered  to.  As  an  Acquisition  Category  III 
level  program,  DRRS  will  be  managed  from  an  acquisition  perspective  by  the 
Component  Acquisition  Executive  (CAE)  within  USD(P&R).  The  USD  (P&R)  CAE  is 
already  working  with  the  program  to  ensure  that  they  become  fully  compliant  with  all 
acquisition  requirements.  This  is  consistent  with  tiered  accountability.  The  CAE  will 
certify  to  the  HRM  IRB  and  the  DCMO  of  compliance  prior  to  submission  of  future 
certification  requests.  Finally,  the  current  DEXCOM  governance  process  will  continue  to 
provide  sustained  functional  oversight  of  the  DRRS  program.  Issues  that  arise  in  any  of 
these  areas  will  be  elevated  for  review  as  appropriate. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 
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The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 
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